Quality and Bias in Meditation Research: Blinding, Expectation and Publication Bias

Good research on meditation exists, but it is harder to do than a drug trial. This page explains the main problems, what researchers have measured about them, and a short checklist for reading any study or news story. It is information only. None of this means meditation is useless; it means claims need careful reading. For results themselves, see does meditation work?.

Why this field has special problems

A group of mindfulness researchers wrote in 2018 that misinformation and poor methodology in past studies may lead the public to be harmed, misled and disappointed. [1] Their review looked at defining mindfulness, measuring it, training, possible adverse effects and brain imaging. [1] Another methods paper described the challenges of research designs in which true double-blinding is not possible, the choice of control conditions, and what can and cannot be inferred from self-report. [2]

Problem 1: Blinding and expectation

In a drug trial, neither patient nor doctor knows who gets the real pill. In a meditation trial, participants know whether they are meditating, and teachers know what they teach. [2] If people expect to feel calmer, they may report feeling calmer. This matters most for self-reported outcomes such as stress or mood, and less for objective measures. Some trials reduce the problem with a blinded assessor: in a trial comparing MBSR with escitalopram, outcomes were assessed by a blinded clinical interviewer. [3] Participants still knew which treatment they had.

Problem 2: Weak comparison groups

Many trials compare mindfulness with a waitlist. A waitlist cannot account for the attention, group support and structure of a course. In a review of 136 trials, mindfulness programs did better than no intervention, but against general active controls only depression clearly improved, and there was no significant evidence of beating specific active treatments. [4] In another review, effects were d = 0.55 against no treatment and d = 0.23 against specific active controls. [5] Better comparison groups give smaller effects.

Problem 3: Selective reporting and publication bias

An analysis of 124 published mindfulness trials found 87% reported at least one positive outcome in the abstract, 1.6 times more than expected from the average effect size. [6] Of 21 trial registrations, 13 were still unpublished 30 months after completion, and none specified a single primary outcome with its time of assessment. [6] The authors concluded that the share of significant trials may overstate what would occur in practice. [6] A meta-analysis of brain structure studies also called publication bias a strong concern. [7]

Problem 4: Small studies and failed replication

Early positive findings can fade. A small controlled study found gray matter increases after MBSR, [8] but two combined randomized trials with 218 people found no such change. [9] The authors described earlier studies as widely referenced despite major methodological limitations. [9]

Problem 5: Vague definitions

"Mindfulness" and "meditation" are used for very different practices. A review noted the difficulty of defining mindfulness and the proper scope of research into it. [1] A trial of an 8-week group course tells us little about a 5-minute app session. See how researchers define meditation.

Problem 6: Harms are not well studied

NCCIH says few studies have examined potentially harmful effects, so definite safety statements are not possible. [10] It describes a 2020 review of 83 studies (6,703 participants) in which about 8% of participants had a negative effect, similar to the share reported for psychological therapies, most often anxiety and depression. [10] In a survey of 1,232 regular meditators, 25.6% reported unpleasant experiences they thought might be caused by meditation. [11] Surveys like this cannot give a true rate, because they depend on who answered. See difficulties and distressing experiences.

A five-question checklist

1. What was the comparison group?

   Waitlist results are not the same as results against an active treatment [5]

2. How many people, and who?

   Small studies often fail to replicate [9]

3. Was the outcome self-reported or objective?

   Self-report is affected by expectation [2]

4. Was the trial registered, with a clear main outcome?

   Unregistered or unclear outcomes allow selective reporting [6]

5. Who paid or has an interest?

   Read the disclosure statements, as [5] shows

What the research still supports

NCCIH says much of the research has been preliminary or not rigorous, but reviews with active controls still found moderate evidence of small improvements for anxiety, depression and pain. [10][12] Reading carefully means holding both ideas. See also meta-analyses of mindfulness trials and criticism of mindfulness.