Publication Bias and Positive-Result Concerns in Chinese Trials

What publication bias means

Publication bias happens when studies with favorable results are more likely to be written up, accepted or found than studies with disappointing results. If it is strong, a pile of published trials can make a treatment look better than it is. It can affect any field. This page covers what researchers have measured for trials of Traditional Chinese Medicine (TCM) and trials from China more generally, and where the evidence stops.

The 1998 finding that started the conversation

A systematic review published in 1998 looked at trials listed in Medline up to 1995. [1] In acupuncture trials, all trials from China, Japan, Hong Kong and Taiwan were positive. [1] For treatments other than acupuncture, 75% of trials published in England favored the test treatment, compared with 99% from China, 89% from Japan, 97% from Russia/USSR and 95% from Taiwan, and no trial published in China or Russia/USSR found a test treatment to be ineffective. [1] The authors wrote that publication bias is a possible explanation, and advised reviewers to consider how to manage data from these countries. [1]

Note the careful wording: "possible explanation." The study could not show why the results were so uniformly positive, and it covered all kinds of treatments, not only TCM.

Later measurements for Chinese herbal medicine

• Small trials and language. A comparison of 136 placebo-controlled Chinese herbal trials with 136 matched conventional trials found that smaller trials showed more beneficial effects than larger ones in every group, a pattern that can signal bias. [2] Herbal trials published in Chinese reported considerably larger effects than those published in English (adjusted ratio of odds ratios 0.29, 95% confidence interval 0.17 to 0.52). [2] The authors concluded biases occur in both herbal and conventional placebo-controlled trials but may be most pronounced in herbal trials in Chinese-language journals. [2]
• Are they truly randomized? In a 2009 study, 3,137 apparent randomized trials from a Chinese database were checked by interviewing authors; only 6.8% had used accepted randomization methods. [3] The authors noted these reports mostly present positive results, and found no significant difference in authenticity between traditional and conventional intervention trials. [3]
• Reporting and conduct. A review of 307 randomized trials conducted in China found nearly 44% reported a zero dropout rate; the authors said conduct cannot be directly inferred from reporting. [4] This is a flag for follow-up, not proof of wrongdoing.

Why it also works the other way

Excluding Chinese-language trials has its own risk. One study found that 96.64% (115 of 119) of eligible trials of Chinese herbal medicine for coronary artery disease could be identified only from Chinese databases. [5] In a random sample of 80 Cochrane reviews on TCM, Chinese databases had been searched in only 43 (53.75%). [5] A separate overview of 58 Cochrane reviews of Chinese herbal medicine found 10 that did not search any Chinese database. [6] Reviewers face a trade-off: leave Chinese trials out and lose most of the evidence, or include them and take on the quality concerns above. The best reviews include them, rate each trial's risk of bias, and report the uncertainty.

Summary table

What this means for you

• A positive-sounding TCM headline from a single small trial deserves caution, especially if the trial was small, from a single center, or compares a formula with nothing.
• The right response is not to dismiss the whole field. The same sources show that good trials exist and that reporting standards are improving; see Quality of TCM Clinical Trials.
• Look for systematic reviews that rate quality, such as those summarized in What Cochrane Reviews Say About Chinese Herbal Medicine, and use How to Read a TCM Study.
• For the big-picture answer, see Does Traditional Chinese Medicine Work?