How a Tiny Number Warps Scientific Discovery
Imagine a world where traffic lights turned green only if your speedometer hit exactly 60.000 mph. Chaos, right? Strangely, something similar happens in science, thanks to a tiny, arbitrary number: 0.05. This ubiquitous threshold for "statistical significance" isn't just a line in the sand; it's actively distorting the map of scientific findings, creating a bizarre pothole in the landscape of reported results. Let's explore how this happens and why it matters for the trustworthiness of science itself.
At the heart of many scientific claims lies the p-value. Simply put, it's the probability of seeing your experimental results (or something more extreme) if there was actually no real effect (the null hypothesis). A low p-value suggests your results are unlikely to be just random noise.
Decades ago, statistician Ronald Fisher suggested p < 0.05 as a handy, though arbitrary, threshold for indicating "surprising" results worthy of note. Fast forward to today, and "p < 0.05" has become the golden ticket for publication. Journals crave "significant" findings, and researchers need publications. This intense pressure creates a powerful incentive structure.
Here's the catch: Nature doesn't care about our thresholds. Real effects produce a smooth distribution of p-values. But what happens when human behavior and publication bias intervene?
Studies finding "non-significant" results (p ≥ 0.05) often vanish into researchers' file drawers, unpublished. They aren't "exciting" enough.
The manipulation of data or analysis choices after seeing results to nudge a p-value just below 0.05. This includes trying different tests, removing outliers, or selectively reporting.
The combined effect of selective reporting and p-hacking leaves a distinct fingerprint on the distribution of p-values that actually get published:
To see this phenomenon in action, let's look at a crucial 2017 study by Robert Ulrich and John Miller published in Psychological Science.
Analyzed p-values from 12,000 psychology journal articles, extracting over 250,000 individual p-values using automated and manual methods.
Categorized p-values based on whether they supported the paper's main conclusion, then plotted their frequency distribution around the 0.05 threshold.
The findings were stark:
| p-Value Bin | Observed Frequency | Expected Frequency |
|---|---|---|
| 0.040 < p ≤ 0.045 | Very High | Moderate |
| 0.045 < p ≤ 0.050 | Extremely High (Peak) | Moderate |
| 0.050 < p ≤ 0.055 | Very Low (Cliff) | Moderate |
| 0.055 < p ≤ 0.060 | Very Low | Moderate |
"The frequency of p-values reported in the crucial bin just below 0.05 (0.045-0.05) was vastly higher than in the bins immediately above it (0.05-0.055 and 0.055-0.06), by more than a factor of 10. This stark discontinuity is highly improbable under honest reporting."
| Bin 0.045-0.05 vs. Bin 0.040-0.045 | ~1.5x |
| Bin 0.045-0.05 vs. Bin 0.050-0.055 | > 10x |
| Bin 0.045-0.05 vs. Bin 0.055-0.060 | > 10x |
| Psychology | High |
| Social Sciences | High |
| Biomedicine | Moderate to High |
| Physical Sciences | Lower (but not absent) |
This unnatural distribution isn't just a statistical quirk; it has serious consequences:
Many results barely under p=0.05 are likely false positives, boosted by selective reporting or p-hacking.
The published record becomes skewed, over-representing weak effects and under-representing null results.
Researchers waste time and money trying to build upon findings that were statistical flukes.
Undermines confidence in scientific research when these practices become known.
Researchers striving for rigor use several tools to mitigate these issues:
Publicly detailing hypotheses, methods, & analysis plan before data collection. Prevents p-hacking by locking in the plan.
Sharing raw data and analysis scripts allows others to verify results and check for p-hacking.
Reporting the magnitude and precision of effects, not just significance (p-value). Provides more meaningful information.
An alternative framework focusing on evidence strength for hypotheses, less reliant on arbitrary thresholds.
The gold standard: independently repeating experiments to see if results hold. Directly addresses false positives.
Large-scale collaborations reduce individual incentives for questionable practices.
The p-value bump is a symptom of a deeper problem: our over-reliance on a single, arbitrary threshold. So, what's changing?
Some fields advocate for p < 0.005 for claiming new discoveries to reduce false positives.
Many journals discourage dichotomous language, urging precise p-values and emphasis on effect sizes.
Promoting transparency, valuing replication, and rewarding robust methods over flashy results.
The tiny threshold of 0.05 has cast a surprisingly long shadow, warping the very shape of published scientific knowledge. By recognizing its unintended influence – the tell-tale p-value pothole – and adopting more robust and transparent practices, scientists are working to build a smoother, more reliable road to discovery. The goal isn't to abandon p-values, but to use them wisely, as one tool among many in the quest for genuine understanding.