My day job is researching and sharing ideas--hopefully good ones. You may have heard of statistical-significance testing in science. If the difference between two sample means is “significant” that just means it’s probably real and not random chance. “Statistical significance” isn’t the same as practical significance, or plain-English significance.
A new medicine may reduce the length of hospitalization compared to the leading drug. But if the two means are 6.10 days versus 6.14, the average benefit was just one hour. In a large sample, this difference will be statistically significant. But practically, the two drugs are equal. To tout the new one as superior would be misleading. And what if it costs twice as much, or has worse side effects?
For realistic differences we turn to estimates of effect size, which most people have never heard of. Effect size reveals practical significance, or magnitude. An example of a large effect is in the strength of men versus women. The men’s world-record dead lift is 501 kilograms by Icelander Hafþór Björnsson. The women’s record is 325 kilos by England’s Lucy Underdown. Björnsson lifted 54 percent more weight. Sex differences in strength are both statistically significant and large, with little overlap. In contrast, some studies report differences that are so small as to be trivial.
The late psychologist Frank Schmidt chastised the “severe deficiencies” of relying on statistical-significance alone. He said many studies never report their effect sizes because they are so small. Publishing useless findings holds back the accumulation of scientific knowledge.
Science and social science bring huge value to our lives. But if someone tells you about a statistically-significant finding, ask “how large was the effect size?”
I’m Jeff Gentry Hafþór: “Half-thor”
Best reference:
Schmidt, F.L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1 (2), 115-129.
Dumb Ideas that Changed the World copyright 2025 by Jeff Gentry. All rights reserved.
