Statistical vs Clinical Significance

Two meanings of "significant"

In everyday speech, "significant" means important. In statistics it has a narrow technical meaning, and the two are often confused in health news. A result can be statistically significant yet too small to matter to anyone, or fail to reach significance in a study that was simply too small to tell. This page explains the difference. It is education, not medical advice.

What a p-value is

A p-value is a calculation about how unusual the data would be if a particular assumption (usually "no difference") and all the other statistical assumptions were true. It is not the probability that the hypothesis is true. A statistical review of common misinterpretations says the p-value assumes the test hypothesis is true and "is not a hypothesis probability." [1]

The same review explains that a small p-value simply flags the data as unusual under the assumptions used, and it may be small because of random error or because some assumption was violated, not necessarily because the tested effect is real. [1] Many journals and reviews use 0.05 as a cutoff, but that threshold is arbitrary and became common largely through historical habit. [2]

Statistically significant is not the same as important

The review is direct on this: a statistically significant result does not mean that a scientifically or clinically important relationship was found. When a study is large, very minor effects can produce a statistically significant test, and the way the data are unusual may be of no clinical interest. [1]

The reverse also holds. Lack of statistical significance does not mean the effect is small. In a small study, even a large effect can be "drowned in noise" and fail to reach significance. [1] The Cochrane Handbook warns that review authors often mistake a lack of evidence of effect for evidence of no effect. [2]

What to look at instead: effect size and the range

The Cochrane Handbook advises focusing interpretation on estimates of effect and their confidence intervals instead of labeling results "significant" or "non-significant," and reporting the exact p-value alongside the confidence interval. [2] A confidence interval gives a range of effect sizes that fit the data reasonably well, which lets you ask whether the range includes changes that would matter.

What counts as clinically meaningful

Whether a change matters depends on the outcome and the person. Researchers use the minimal important difference (MID), the smallest change in a score that patients perceive as important, to give results context. The Cochrane Handbook says knowing the MID can greatly help interpretation, and recommends stating it when known. [2]

MIDs are not simple. A systematic review of 80 articles on MIDs for quality-of-life measures found little consistency in how they were calculated: only 54% of the studies used both anchor-based and distribution-based methods, and the reviewers said an MID should be interpreted with caution. [3] So a claim that a result "exceeded the clinically meaningful threshold" is only as good as the method behind that threshold.

Reading a health claim

• Ask for the size of the change, not only "significant."
• Ask whether the outcome is something you would notice, or only a lab value; see surrogate vs patient-important outcomes and biomarkers vs clinical outcomes.
• Ask whether the result is shown in percentages without the starting risk; see absolute vs relative effects.
• Ask whether other studies agree; see systematic reviews and meta-analyses explained.
• Be careful with "clinically proven" based on one small study; see claim check: clinically proven.

Decisions about your own care belong with a clinician who knows your history.