Studying Traditional Medicine in Clinical Trials: Whole Systems, Individualization and Blinding

The standard trial and where it strains

The usual test of a treatment is a randomized controlled trial: people are assigned by chance to the treatment or a comparison, ideally without knowing which, and outcomes are measured. This works best for a single, fixed treatment such as a tablet with one set dose.

Many traditional systems are built differently. This page explains the specific difficulties, using Chinese herbal medicine as the best-documented example, and how researchers try to meet them. It does not mean traditional approaches cannot be tested, and it does not excuse a lack of evidence. A treatment that is hard to study is still unproven until it has been studied. For the basics of trials, see understanding evidence in complementary health.

Difficulty 1: Treatment is individualized

Complementary and traditional systems often combine several approaches to give individualized treatment. [1] A workshop of researchers and practitioners listed treatment individualization, problems of diagnosis, patient-practitioner interaction, varying therapeutic contexts and patient-determined outcome values as key issues. [2]

In Chinese herbal medicine, for example, the formula can depend on a traditional "pattern" (zheng) identified by the practitioner. A review of a Chinese herbal prescription for stroke listed shortcomings in the traditional diagnostic process: lack of standard terms, disagreement about pattern differentiation, and neglect of matching the formula to the pattern. [3] If two practitioners would diagnose the same person differently, it is hard to describe exactly what was tested.

A traditional pattern is a framework from that tradition, not a proven biological category. A trial can still test whether people given an individualized approach do better than a comparison group on a measured outcome.

Difficulty 2: Whole systems versus single parts

Some researchers argue that classical randomized trials are limited in their ability to address the unique healing theory and context of whole systems of care, and propose mixed methods with a range of outcome measures. [1] A comparison of four approaches to evaluating complex health systems found agreement on using mixed methods, including randomized trials (both explanatory and pragmatic) and qualitative approaches. [4]

In plain terms: a trial that isolates one herb answers a different question than a trial of a whole course of care. Both are legitimate. Neither should be described as proving the other.

Difficulty 3: Blinding and placebos

Blinding means participants do not know which treatment they get. For herbs, it is hard to make a convincing placebo. The stroke review noted that it is difficult to design credible herbal placebos with similar appearance, smells and tastes to the experimental formula. [3] It also noted that suspicion of placebo and unwillingness to stop other herbs make recruitment harder. [3]

Why this matters: how a trial is run affects the result. In the Cochrane review of placebo trials, larger placebo effects were linked with physical placebos such as sham acupuncture, with patient-reported outcomes, and with small trials. [5] Trials that cannot blind well, or that rely on self-reported symptoms, are more open to expectation effects.

Difficulty 4: What exactly is in the product

A trial result applies to the product tested. Quality of starting material matters: WHO notes that wild-collected plants may be contaminated by other species or plant parts through misidentification or adulteration. [6] A trial using one extract tells us little about a different product with different ingredients.

Difficulty 5: Poor reporting

Reports of herbal trials are often incomplete. The CONSORT extension for Chinese herbal medicine formulas states that the general reporting quality of such trials is disappointing and adds items on the traditional pattern and the features of the formula. [7] A Cochrane review of herbal medicine for low-back pain made a similar point, noting poor completeness of reporting, and that trialists should refer to the CONSORT extension for herbal trials. [8]

An example of the result

A systematic review of a Chinese herbal prescription for acute ischemic stroke found 8 trials with 601 people, judged to be at high risk of bias. [3] The authors called the evidence positive but weak because of poor method and few trials. [3] This is the typical pattern: some signal, not enough quality to be confident.

How researchers respond

Reading a trial claim about traditional medicine

• What exactly was tested: one herb, a fixed formula, an individualized approach, or a whole system?
• Compared with what: placebo, usual care, nothing?
• Was it blinded, and was the outcome self-reported?
• How many people, and was the trial registered and reported fully?
• Is the product you can buy the same as the one tested?

For the broader picture, see historical use vs clinical evidence, why long history is not proof and therapies research and evidence. Talk with a clinician before changing any treatment; do not stop prescribed medicine on the basis of a single study.