Effect Sizes and Confidence Intervals in Plain Language

Two questions a result must answer

When a study reports that a treatment "worked," two separate questions matter: how big was the difference, and how sure are we about that number? The first is the effect size. The second is shown by a confidence interval. A p-value, the number many headlines lean on, answers neither question well. This page is a plain-language guide to reading these numbers; it is general education and not advice about any treatment.

Effect size: how big was the difference?

An effect size is the magnitude of the difference between groups [1]. It can be absolute (for example, a difference of 15 points on a 50-point test) or standardized, where the difference is divided by the spread (standard deviation) of the scores, which turns it into standard-deviation units [1]. A common standardized measure is Cohen's d; Cohen's rough labels call 0.2 small, 0.5 medium and 0.8 large, but these ballpark labels ignore context such as how accurate the measure is, and should be a guide only [1].

Why not just use the p-value? Statistical significance depends on both the effect size and the sample size [1]. With a sufficiently large sample, a test will almost always show a significant difference unless the effect is exactly zero, so very small differences can be significant yet meaningless [1]. The example in the paper: the Physicians' Health Study of aspirin followed more than 22,000 people for an average of 5 years and found a highly statistically significant reduction in heart attacks (P < .00001), but the absolute difference was only 0.77%; later studies found even smaller effects and the aspirin recommendation was since modified [1]. See Statistical vs Clinical Significance.

Relative and absolute: the same result, two sizes

The Cochrane Handbook gives a clear example. A treatment that cuts the relative risk of death by 25% and one that cuts risk by 1 percentage point can describe the same benefit: a fall from 4% to 3% [2]. The first is a relative reduction, the second an absolute one [2].

• Risk ratio: a risk ratio of 0.75 equals a 25% relative risk reduction [2].
• Number needed to treat (NNT): the expected number of people who need to receive the experimental treatment rather than the comparator for one additional person to avoid (or have) an event. It is an expected value, not a promise that one in every ten people benefits, and it compares a treatment with a specific comparator [2].

When a news item says "cuts risk by 50%," ask: from what to what? See Absolute vs Relative Effects.

Confidence interval: how precise is the estimate?

Results are reported as a best estimate (the point estimate) with a confidence interval, for example, an odds ratio of 0.75 with a 95% confidence interval of 0.70 to 0.80 [2]. The interval describes the uncertainty in the estimate [2]. The Handbook's guide to reading width [2]:

For ratios, 1.0 means no difference; an interval that includes 1.0 is compatible with no effect and with an effect in either direction. Larger studies tend to give narrower intervals [2].

A worked example

A Cochrane review of placebo versus no treatment found a pooled relative risk of 0.93 (95% CI 0.88 to 0.99) for yes-or-no outcomes [3]. In the Handbook's terms, 0.93 corresponds to about a 7% relative reduction in risk. The interval's upper end sits close to 1.0, so the data are consistent with a very small effect. For continuous outcomes the standardized effect was -0.23 (95% CI -0.28 to -0.17), "small" under Cohen's rough labels [3][1]. The authors concluded placebo has no important clinical effects in general, while noting wide uncertainty in individual conditions [3].

Common misreadings

• "There is a 95% chance the true effect is inside this interval." Strictly, this is wrong for a single interval. The 95% describes how often intervals calculated from very many similar studies would contain the true effect if the assumptions were correct [4]. The Handbook accepts it only as a loose guide [2].
• "A very small p-value proves the treatment works." A small p-value can arise from a false hypothesis, but also from broken study protocols or from results chosen because they were small [4].
• "A wide interval that includes no effect means no effect." It means the study could not rule an effect in or out; the Handbook cautions against relying on significance thresholds and recommends reporting the interval with the exact p-value [2].
• "An interval that excludes zero is more precise." Precision is the width of the interval, not whether it crosses the null value [4].
• "Statistically significant means important." Importance needs a yardstick: the smallest change patients notice, called the minimal important difference (for one breathing questionnaire scored 1 to 7, 0.5 is an established example) [2].

A short checklist

1. What was the intervention and the comparator, and in whom?
2. What was the outcome: a patient-relevant result, or a lab marker? See Biomarker Changes vs Meaningful Health Outcomes.
3. What is the effect size, in absolute terms?
4. How wide is the confidence interval?
5. Was the study designed to limit bias? See Randomization, Blinding and Placebo.