event
Thursday 02 Mar 2017: Statistical Science Seminar: Quantifying Discovery in Astro/Particle Physics: Frequentist and Bayesian Perspectives
David A. van Dyk - Imperial College
H101 15:00-16:30
The question of how best to compare models and select among them has bedeviled statisticians, particularly Bayesian statisticians, for decades. The difficulties with interpreting p-values are well known among Bayesians and non-Bayesians alike. Unfortunately, the strong dependence on the choice of prior distribution of the most prominent fully Bayesian alternative , the Bayes Factor, has limited its popularity in practice. In this talk, we explore a class of non-standard model comparison problems that are important in astrophysics and high-energy physics. The search for the Higgs boson, for example, involved quantifying evidence for a narrow component added to a diffuse background distribution. The added component corresponds to the Higgs mass distribution, accounting for instrumental effects, and cannot be negative. Thus, not only is the null distribution on the boundary of the parameter space, but the location of the added component is unidentifiable under the null. Because many researchers have a strong preference for frequency-based statistical methods, they employ a sequence of likelihood ratio tests on a grid of possible null values of the unidentifiable location parameter. We compare Bonferroni and a Markov bounds on the resulting p-value, both of which are popular methods for correcting for the multiple testing inherent in this procedure. We then suggest a Bayesian strategy that employs a prior distribution on the location parameter and show how this prior automatically corrects for the multiple testing. The Bayesian procedure is significantly more conservative in that it avoids the well-known tilt of p-values toward the alternative when testing a precise null hypothesis. Finally, we discuss the circumstance under which the dependence of the Bayes Factor can be interpreted as a natural correction for multiple testing.
