Avoiding Selective Interpretation

The common traps

Confirmation bias. Favoring information that supports what you already think. In a systematic review of 20 studies with 6,810 physicians, all studies found at least one cognitive bias or personality trait affecting the doctors, and overconfidence, anchoring and availability biases were linked to diagnostic inaccuracies. [1] If trained clinicians are affected, so are the rest of us. That review studied physicians, not people tracking themselves, so it shows the bias is common but not how large it is for personal experiments.

Illusions of causality. These occur when people develop the belief that there is a causal connection between two events that are actually unrelated. [2] A good week after a new routine feels like proof, even when the week was good for other reasons.

Cherry-picking. Quoting the best night, the best week or the one metric that moved. With dozens of numbers each day, some will improve by chance. Researchers note that a finding is less likely to be true when there is a greater number and lesser preselection of tested relationships, and greater flexibility in designs, definitions, outcomes and analysis. [3] Personal tracking is exactly that kind of setting: many numbers, nothing chosen in advance.

Regression to the mean. Unusually large or small measurements tend to be followed by measurements closer to the mean. [4] If you start a new habit after a terrible week, things may improve anyway. See regression to the mean in personal data.

Expectation. The placebo effect is a beneficial health outcome resulting from a person's anticipation that an intervention will help. [5] Expecting a gadget or habit to work can change how you feel and how you read the data. See placebo and expectancy effects.

Habits that help

These do not remove bias. They make it easier to see.

1. Write the question and the one main measure before you look at any data. See defining a clear question and selecting a meaningful outcome.
2. Keep a note of the days that did not fit, not only the ones that did.
3. Record the context such as poor sleep, illness, travel or a stressful week. See recording context and confounders.
4. Compare like with like, for example the same weekdays, rather than your best period against your worst.
5. Ask what else could explain it before accepting your first explanation.
6. Decide in advance what would change your mind, such as "if my morning energy score is no different after several weeks, I will let this go."
7. Show your notes to someone skeptical and ask them to argue the other side.

Evidence labels used here

Established means supported by consistent, good-quality research. Insufficient evidence means we could not find enough to judge. That people are prone to these biases is established. We found insufficient evidence on how much any particular habit above reduces bias in personal experiments; the list is common-sense practice, not tested treatment.

When it becomes a health concern

If tracking leaves you anxious, checking constantly or seeking treatment for something only a device suggests, see tracking anxiety and compulsive measurement and speak with a clinician. For the bigger picture of what personal data can and cannot show, see N-of-1 research: meaning and limits.