Conceptual

1. Using basic statistical properties of the variance, as well as singlevariable calculus, derive (5.6). In other words, prove that α given by (5.6) does indeed minimize Var( αX + (1 − α ) Y ).

2. We will now derive the probability that a given observation is part of a bootstrap sample. Suppose that we obtain a bootstrap sample from a set of n observations.

   • (a) What is the probability that the first bootstrap observation is not the j th observation from the original sample? Justify your answer.

   • (b) What is the probability that the second bootstrap observation is not the j th observation from the original sample?

   • (c) Argue that the probability that the j th observation is not in the bootstrap sample is (1 − 1 /n ) [n] .

   • (d) When n = 5, what is the probability that the j th observation is in the bootstrap sample?

   • (e) When n = 100, what is the probability that the j th observation is in the bootstrap sample?

   • (f) When n = 10 , 000, what is the probability that the j th observation is in the bootstrap sample?

   • (g) Create a plot that displays, for each integer value of n from 1 to 100 , 000, the probability that the j th observation is in the bootstrap sample. Comment on what you observe.

   • (h) We will now investigate numerically the probability that a bootstrap sample of size n = 100 contains the j th observation. Here j = 4. We first create an array store with values that will subsequently be overwritten using the function np.empty() . We then np.empty()

5.4 Exercises 225

repeatedly create bootstrap samples, and each time we record whether or not the fifth observation is contained in the bootstrap sample.

rng=np.random.default_rng(10)
store=np.empty(10000)
foriinrange(10000):
store[i]=np.sum(rng.choice(100,replace=True)==4)
>0
np.mean(store)

Comment on the results obtained.

1. We now review k -fold cross-validation.

   • (a) Explain how k -fold cross-validation is implemented.

   • (b) What are the advantages and disadvantages of k -fold crossvalidation relative to:

     • i. The validation set approach?

     • ii. LOOCV?

2. Suppose that we use some statistical learning method to make a prediction for the response Y for a particular value of the predictor X . Carefully describe how we might estimate the standard deviation of our prediction.