Conceptual

1. Suppose we test m null hypotheses, all of which are true. We control the Type I error for each null hypothesis at level α . For each subproblem, justify your answer.

   • (a) In total, how many Type I errors do we expect to make?

   • (b) Suppose that the m tests that we perform are independent. What is the family-wise error rate associated with these m tests? Hint: If two events A and B are independent, then Pr( A ∩ B ) = Pr( A ) Pr( B ) .

   • (c) Suppose that m = 2, and that the p -values for the two tests are positively correlated, so that if one is small then the other will tend to be small as well, and if one is large then the other will tend to be large. How does the family-wise error rate associated with these m = 2 tests qualitatively compare to the answer in (b) with m = 2?

Hint: First, suppose that the two p-values are perfectly correlated.

• (d) Suppose again that m = 2, but that now the p -values for the two tests are negatively correlated, so that if one is large then the other will tend to be small. How does the family-wise error rate associated with these m = 2 tests qualitatively compare to the answer in (b) with m = 2?

Hint: First, suppose that whenever one p-value is less than α, then the other will be greater than α. In other words, we can never reject both null hypotheses.

594 13. Multiple Testing

1. Suppose that we test m hypotheses, and control the Type I error for each hypothesis at level α . Assume that all m p -values are independent, and that all null hypotheses are true.

   • (a) Let the random variable Aj equal 1 if the j th null hypothesis is rejected, and 0 otherwise. What is the distribution of Aj ?

   • (b) What is the distribution of[�] [m] j =1 [A][j][?]

   • (c) What is the standard deviation of the number of Type I errors that we will make?

2. Suppose we test m null hypotheses, and control the Type I error for the j th null hypothesis at level αj , for j = 1 , . . . , m . Argue that the family-wise error rate is no greater than[�] [m] j =1 [α][j][.]

Null Hypothesis	p-value
_H_01 _H_02 _H_03 _H_04 _H_05 _H_06 _H_07 _H_08 _H_09 _H_10	0._0011 0.031 0.017 0.32 0.11 0.90 0.07 0.006 0.004 0._0009

TABLE 13.4. p-values for Exercise 4.

1. Suppose we test m = 10 hypotheses, and obtain the p -values shown in Table 13.4.

   • (a) Suppose that we wish to control the Type I error for each null hypothesis at level α = 0 . 05. Which null hypotheses will we reject?

   • (b) Now suppose that we wish to control the FWER at level α = 0 . 05. Which null hypotheses will we reject? Justify your answer.

   • (c) Now suppose that we wish to control the FDR at level q = 0 . 05. Which null hypotheses will we reject? Justify your answer.

   • (d) Now suppose that we wish to control the FDR at level q = 0 . 2. Which null hypotheses will we reject? Justify your answer.

   • (e) Of the null hypotheses rejected at FDR level q = 0 . 2, approximately how many are false positives? Justify your answer.

2. For this problem, you will make up p -values that lead to a certain number of rejections using the Bonferroni and Holm procedures.

   • (a) Give an example of five p -values (i.e. five numbers between 0 and 1 which, for the purpose of this problem, we will interpret as p - values) for which both Bonferroni’s method and Holm’s method

13.7 Exercises 595

reject exactly one null hypothesis when controlling the FWER at level 0 . 1.

• (b) Now give an example of five p -values for which Bonferroni rejects one null hypothesis and Holm rejects more than one null hypothesis at level 0 . 1.

1. For each of the three panels in Figure 13.3, answer the following questions:

   • (a) How many false positives, false negatives, true positives, true negatives, Type I errors, and Type II errors result from applying the Bonferroni procedure to control the FWER at level α = 0 . 05?

   • (b) How many false positives, false negatives, true positives, true negatives, Type I errors, and Type II errors result from applying the Holm procedure to control the FWER at level α = 0 . 05?

   • (c) What is the false discovery proportion associated with using the Bonferroni procedure to control the FWER at level α = 0 . 05?

   • (d) What is the false discovery proportion associated with using the Holm procedure to control the FWER at level α = 0 . 05?

   • (e) How would the answers to (a) and (c) change if we instead used the Bonferroni procedure to control the FWER at level α = 0 . 001?