How to Read a Wearable Validation Study

Step 1: What was the device compared with?

The comparison is called the reference or criterion measure. For sleep, the standard comparison is polysomnography (PSG), an overnight lab test. A published framework for testing sleep trackers describes comparing them with gold-standard PSG or other reference methods. [1] The right reference depends on the measure being tested. A weak reference makes the whole study weak. See device vs clinical reference.

Step 2: Who was tested, and under what conditions?

Expert recommendations for wearable heart-rate validation propose six domains: the target population, the criterion measure, the index measure (the device), testing conditions, data processing and statistical analysis. [2] Use them as a checklist:

• Population: results from one group of people may not carry over to another; see wearable validation in different populations.
• Conditions: resting, walking and hard exercise can give different results, and so can how the device is worn; see fit, placement and skin factors.
• Size and setting: a small study over one night in a lab may not describe your daily life. For example, one sleep study enrolled 35 healthy adults aged 20 to 50 in a single-night inpatient setting. [3]

Step 3: Which exact device?

Results belong to the exact model, version and software tested. A newer or older version may behave differently, so do not carry a result across models. See how to find validation studies and regulatory records.

Step 4: Understand the numbers

Bland-Altman plots are a standard way to show agreement. Correlation studies the relationship between one variable and another, not the differences, and is not recommended for assessing whether two methods are comparable. [4] A Bland-Altman analysis only defines the interval of agreement. It does not say whether those limits are acceptable; acceptable limits must be decided beforehand from clinical need or other goals. [4] Sleep-tracker guidelines also name epoch-by-epoch analysis, which checks agreement one short time slice at a time. [1]

Step 5: One device, different results by measure

A device can do well on one measure and poorly on another. Examples from reviews and studies:

• In a meta-analysis of 56 studies of one smartwatch brand, energy expenditure subgroups all exceeded a 10% MAPE validity threshold, while none of the heart-rate subgroups did. [5]
• A systematic review of wrist-worn activity trackers found energy-expenditure MAPE above 30% for all brands. [6]
• In a study of three sleep devices, sensitivity for sleep versus wake was at least 95%, but for sleep stages it ranged from 50 to 86%. [3]

These are results for specific devices and populations at the time of the studies; they do not rank brands and do not describe current models.

Step 6: Accuracy is not benefit

Even a very accurate reading does not prove that wearing a device improves health. That needs a separate type of study that follows outcomes. Be cautious of proprietary scores, which are calculated by a company and may have no clinical meaning; see wearables and health trackers. The American Academy of Sleep Medicine states that, given lack of validation and FDA clearance, consumer sleep technologies cannot be used for diagnosis or treatment of sleep disorders at this time. [7]

Evidence labels used here

Established means supported by consistent, good-quality research. Mixed means results differ by measure, device or setting. Wearable accuracy is mixed: it varies by measure, as above.