Life After P-Hacking

This paper considers how a commitment to not publishing p-hacked results will change the lives of individual scientists and products of science. We call this “Life After P-Hacking.” We discuss four lessons. Lesson #1 We have to really start caring about statistical power. The freedom to engage in p-hacking created an environment in which researchers were able to get many of their studies to yield statistically significant results despite dramatically underpowering those studies (or studying truly null effects). “Life after p-hacking” means that conducting underpowered studies will now come with significant costs to the individual researcher: (1) by definition, many underpowered studies will yield null effects, and (2) researchers will not know why they did not work – whether the null effects represent a false hypothesis (a truly null effect) or a false-negative. This makes it difficult to learn and difficult to publish, a combination of costs that is unlikely to be sustainable for the typical researcher. The only way forward, then, is to really start caring about statistical power – to make sure our studies are properly powered at, say, 80%.