The pointlessness of the point null hypothesis test##
Thesis: Point null hypothesis tests (PNHT) are probably not worth conducting. They do not seem to tell us much about whether there is a causal relationship between variables of interest and it is hard to see why else a researcher would use this test.
Subheadings
Intro
What PNHT is
- Definition of a point null and why I’m focusing on it
- Definition of p-values
- Summary of similar procedures (e.g. Bayes’ factors, your group sequential testing procedure for controlled trials)
- Why PNHT is suspect
What it is not (and what it shouldn’t be)
- Tests of point nulls probably aren’t equivalent to tests of small-interval nulls (reply to Meehl 1997 and Rindskopf 1997)
- The point null isn’t the ‘probability that the samples were drawn from different populations’ (reply to Hagen 1997)
- It doesn’t matter that failing to reject point null means shortest confidence interval contains null value in common cases (reply to various authors)
The ‘superpopulation argument’ for PNHT
- The point nil (not null) hypothesis might be interpreted as the prediction that there is no causal relationship between variables of interest
- Summary of superpopulation modelling
- PNHT may be a test of proposition that the sampling distribution of the superpopulation follows the point null distribution
- Even Bayesians might like this solution (Berger and Sellke, Casella and Berger)
Why the argument probably doesn’t work
- Issues with superpopulation modelling (large, unknown variance)
- The most common class of PNHT (tests generating p-values) would overestimate the evidence against the superpopulation point null
- The prior probability of the superpopulation point null isn’t in the right range enough of the time to justify the use of point p-values (returning to Berger and Sellke, Casella and Berger)
- And why PNHT procedures other than those involving p-values might face similar problems
Conclusion
- What (if anything) should replace PNHT?
- What this means for results based on PNHT