P-values and P-hacking

1. ASA Statement on Statistical Significance and P-values
2. Problems with P-values
3. Overfitting
4. Comments from Norm Matloff
5. Podcasts on P-values and P-hacking
6. Videos on P-values and P-hacking
7. Links to Related Topics

• ASA Releases Statement on Statistical Significance and P-values - American Statistical Association, March 7, 2016 - Provides Principles to Improve the Conduct and Interpretation of Quantitative Science.

• The Significant Problem of P Values - Scientific American, October 1, 2019 - Standard scientific methods are under fire. Will anything change?
• Big little lies: a compendium and simulation of p-hacking strategies - Royal Society Publishing, February 8, 2023 - Typically, p-hacking is defined as a compound of strategies targeted at rendering non-significant hypothesis testing results significant.

• Multiple Regression Analysis: Use Adjusted R-Squared and Predicted R-Squared to Include the Correct Number of Variables - Minitab Blog Editor, June 13, 2013 - In this post, we'll look at why you should resist the urge to add too many predictors to a regression model, and how the adjusted R-squared and predicted R-squared can help!
• The Danger of Overfitting Regression Models - Minitab Blog Editor, September 3, 2015 - An overfit model can cause the regression coefficients, p-values, and R-squared to be misleading.
• Overfitting in Regression Models - The Analysis Factor, September 19, 2023 - Harrell describes a rule of thumb to avoid overfitting of a minimum of 10 observations per regression parameter in the model.

• After 150 years, the ASA says No to P-values - Norm Matloff, March 7, 2016 - A number of us have been pointing out for decades that p-values are at best underinformative and often misleading.
• Further Comments on the ASA Manifesto - Norm Matloff, March 9, 2016 - I'll respond in the current post, and will devote most of it to what I believe are the proper alternatives.
• P-values: The Continuing Saga - Norm Matloff, March 10, 2016 - I highly recommend the blog post by Yoav Benjamini and Tal Galili in defense of (carefully used) p-values.
• Any Forward Progress on P-values? - Norm Matloff, February 9, 2017 - But not much seems to have changed in the professions since then, as was shown rather dramatically last Saturday.
• Clearing the Confusion: the Case Against Significance Tests - Norm Matloff, March 23, 2023 - Then what changed? Among other things, a paper by John Ionnidis, provocatively titled, "Why Most Published Research Findings Are False," shook things up .
• Problems with P-values - Norm Matloff, September 28, 2023 - In 2016, the American Statistical Association released its first-ever position paper, to warn of the problems of significance testing and "p-values."

• E12: P-values - Statistically Insignificant, February 2, 2022 - Part 1 of our two-part series on the infamous p-values. In this episode we go through what p-values are and are not, in the context of a hypothesis test.
• E13: P-Hacking - Statistically Insignificant, February 23, 2022 - Our second episode about p-values and their misuse in research. In this episode we explore the manipulation of data to produce statistically significant results, and the material conditions which would lead someone to do that.
• 55. Angelika Stefan: p-hacking, simulations, and Shiny Apps - BJKS podcast, May 1, 2022 - In this conversation, we talk about her preprint 'Big little lies: A Compendium and Simulation of p-Hacking Strategies', which she just uploaded to PsyArXiv.
• 82. Geoff Cumming: p-values, estimation, and meta-analytic thinking - BJKS podcast, November 23, 2023 - In this conversation, we discuss his work on New Statistics: estimation instead of hypothesis testing, meta-analytic thinking, and many related topics.
• Episode 23: Statistical significance - The Studies Show, January 9, 2024 - The FiveThirtyEight online p-hacking tool; The "p-curve" method for detecting p-hacking; How p-hacking is just "overfitting" by another name
• Episode 572: Why Is There So Much Fraud in Academia? - Freakonomics, January 10, 2024 - Last year, an astonishing 10,000 research papers were retracted. We talk to whistleblowers, reformers, and a co-author who got caught up in the chaos. (Part one of two-part series on academic fraud)
• Episode 573: Can Academic Fraud Be Stopped? - Freakonomics, January 17, 2024 - Probably not — the incentives are too strong. Scholarly publishing is a $28 billion global industry, with misconduct at every level. But a few reformers are gaining ground. (Part two of two-part series on academic fraud)

• E12: P-values - Statistically Insignificant, February 2, 2022 - Part 1 of our two-part series on the infamous p-values. In this episode we go through what p-values are and are not, in the context of a hypothesis test.
• E13: P-Hacking - Statistically Insignificant, February 23, 2022 - Our second episode about p-values and their misuse in research. In this episode we explore the manipulation of data to produce statistically significant results, and the material conditions which would lead someone to do that.
• Intro Statistics 9 Dance of the p Values - Geoff Cumming, September 22, 2013 - The simulation illustrates how enormously and disastrously variable the p value is, simply because of sampling variability. Never trust a p value! Use estimation, not NHST!
• Scientific Studies: Last Week Tonight with John Oliver - HBO, May 8, 2016 - John Oliver discusses how and why media outlets so often report untrue or incomplete information as science.
• P-Hacking Your Way To Fame | Part 1 of 6 - What is P-Hacking? - Data Demystified, June 24, 2021 - In 2011, Joe Simmons, Leif Nelson, and Uri Simonsohn published an academic paper titled False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant.
• Gazing into the Abyss of p-Hacking - Angelika Stefan, May 1, 2023 - Dr Stefan discusses how investigating p-hacking at the level of p-hacking strategies may not only provide a better understanding of p-hacking, but also serve as a starting point for developing effective countermeasures.

Javascript Programs
R Shiny Apps and Programs
R and Shiny Resources
Machine Learning Resources
Test Home Page
Windows 7 Tools
Notes from Online Courses
Return to Business Page