Quality of TCM Clinical Trials: Common Problems and Improvements

Why trial quality matters more than trial count

Thousands of randomized trials of Traditional Chinese Medicine (TCM) have been published, but a large pile of weak trials does not add up to strong evidence. A 2022 overview of Cochrane reviews on TCM said so directly, calling for "quality over quantity" in future research. [1] This page explains the common problems and the fixes now in use. It uses the same yardstick for every field: was the trial truly randomized, was it large enough, were people and assessors blinded where possible, and is the treatment described well enough to repeat?

An early picture of the scale

In 1999, researchers hand-searched 28 randomly chosen Chinese TCM journals and found 2,938 randomized trials, with trial numbers doubling every two to three years over the previous 15 years. [2] Volume was never the issue; the question was what each trial could reliably show.

Problem 1: "randomized" does not always mean randomized

A 2009 study searched a large Chinese database for trials on 20 common diseases and found 3,137 reports that appeared to be randomized. [3] When the authors of these reports were interviewed, only 207 (6.8%) had used accepted randomization methods and could be judged authentic randomized trials. [3] This was not a TCM-only finding: the study found no statistically significant difference in authenticity between trials of traditional and conventional interventions. [3] The researchers linked the problem to peer review and to authors' understanding of rigorous trial design.

Problem 2: thin reporting

A review of 307 trials of all kinds conducted in China and published in 2004 found that baseline characteristics, primary outcome and length of follow-up were inadequately reported in a substantial proportion. [4] Fewer than 11% mentioned ethical approval, and only 18.0% adequately discussed informed consent. [4] The authors cautioned that how a trial was conducted cannot be inferred from how it was reported. [4]

TCM-specific reviews show similar gaps:

Problem 3: blinding and the right comparator

Pills can be matched with a look-alike placebo. Needling, cupping, tui na and cooked soups are harder to hide, so trials of those practices are limited by expectation effects. Details are on Placebo, Sham Controls and Blinding in TCM Trials. A trial that compares a formula with a conventional drug, or adds herbs to usual care, answers a different question from a placebo-controlled trial, so the comparator matters when reading results.

Problem 4: few high-quality placebo-controlled trials

A comparison of 136 placebo-controlled Chinese herbal trials, 119 of them published in Chinese, found that only 3 (2%) met the study's definition of high quality (double-blind with adequate sequence generation and concealment), against 10 of 136 (7%) matched conventional trials. [8] Quality was lower in the Chinese-language herbal trials, and the authors concluded few herbal trials of adequate methodology exist. [8] A Cochrane overview of 58 reviews of Chinese herbal medicine found that only 16% of included studies used adequate sequence generation and 7% adequate allocation concealment. [9]

What is improving

• A reporting checklist for herbal formulas. The CONSORT Extension for Chinese Herbal Medicine Formulas 2017 starts from the observation that the general reporting quality of such trials is disappointing. It adds the idea of TCM pattern and the features of herbal formulas to the standard CONSORT checklist, and elaborates seven of the 25 items. [10]
• Checklists for hands-on therapies. Separate reporting standards exist for acupuncture, moxibustion, cupping and tui na (the STRICTA family of checklists). [6]
• Larger trials. A review by Chinese authors reports that, in the past decade, many multicenter randomized acupuncture trials with rigorous designs and adequate sample sizes have appeared in leading international journals. [11] That is the authors' own assessment; the pooled results are covered on the acupuncture pages.