Multiple significance tests and their relation to P-values

Abstract

This thesis is about multiple hypothesis testing and its relation to the P-value. In Chapter 1, the methodologies of hypothesis testing among the three inference schools are reviewed. Jeffreys, Fisher, and Neyman advocated three different approaches for testing by using the posterior probabilities, P-value, and Type I error and Type II error probabilities respectively. In Berger's words ``Each was quite critical of the other approaches." Berger proposed a potential methodological unified conditional frequentist approach for testing. His idea is to follow Fisher in using the P-value to define the strength of evidence in data and to follow Fisher's method of conditioning on strength of evidence; then follow Neyman by computing Type I and Type II error probabilities conditioning on strength of evidence in the data, which equal the objective posterior probabilities of the hypothesis advocated by Jeffreys. Bickis proposed another estimate on calibrating the null and alternative components of the distribution by modeling the set of P-values as a sample from a mixed population composed of a uniform distribution for the null cases and an unknown distribution for the alternatives. For tackling multiplicity, exploiting the empirical distribution of P-values is applied. A variety of density estimators for calibrating posterior probabilities of the null hypothesis given P-values are implemented. Finally, a noninterpolatory and shape-preserving estimator based on B-splines as smoothing functions is proposed and implemented.