Mount Sinai Health System

Shraddha Bhatia

Validation Standards Champion

Global end-user spending on wearable technology is expected to amount to roughly 90 billion USD by 2022; which means that the odds of you reading this article with a wearable device on are pretty high.

Scientists verify usability of instruments by determining their validity, reliability, and accuracy. Validity being the extent to which the instrument used in a given experiment measures exactly what it is meant to measure. Reliability being the extent at which the outcomes are consistent each time the experiment is repeated.

I once heard a researcher claim that it takes roughly ten years for a medical device to go from conception to execution because of how rigorous standardization and testing in healthcare is. Although wearable devices routinely used in the general population now claim to measure physiological data, did you know that they do not undergo exhaustive testing procedures? Not only because of the tremendous challenges, the sheer tedium or high stakes but because of lack of standardization.

INTERLIVE (The Towards Intelligent Health and Well-Being Network of Physical Activity Assessment) a consortium of six European universities and one industrial partner, founded in 2019, identified this void in smartphone and wearable technology validation protocols. The group recognized that while the acceptance of such technology is high, validity of these devices hasn’t been transparently evaluated. They recently released an expert statement in the British Journal of Sports Medicine with a best-practice consumer wearable and smartphone ‘step counter’ validation protocol. As a next step, INTERLIVE plans to provide best-practice recommendations for validation of wearable and smartphone based energy expenditure (EE) and maximal oxygen consumption (VO₂ max) estimations.They argue that since there is no standardized validation protocol for wearable devices or gold standard of measurement to compare against, there is hardly any means for consumers or interested stakeholders to judge if these devices perform satisfactorily and with low measurement errors.

In a 2018 study published in Medicine and Science in Sports and Exercise, heart rate(HR) measurements from eight wearable activity monitors was compared during cycling and resistance exercises. Validity was established using MAPE (mean absolute percent error) which ranged between 4-6.5 percent depending on the exercise. Across all devices, as exercise intensity increased, there was greater underestimation of HR. The study concluded that wearable devices are not the same as medical devices and users should interpret physiological data obtained from these devices with caution. Another validation study monitoring heart rate (HR) and energy expenditure (EE) while sitting or performing light-to-vigorous physical activity with popular wrist-worn wearables found that although the wearables were suitable for valid HR measurements at the intensities tested, error rates increased directly in proportion to higher intensities.Energy expenditure was inaccurate across all the wearables tested.

A systematic review revealed that some wearables that usephotoplethysmography (PPG) may provide a promising alternative solution to ECG for measuring Heart Rate Variability (HRV). But they can only be used as a surrogate for HRV at resting or mild exercise conditions, as their accuracy fades out with increasing exercise load – just like was noted in the above studies for HR. Another reviewinvestigated the diagnostic accuracy of non-12-lead ECG and PPGin detecting atrial fibrillation and found that detection employing automatic analysis techniques may serve as a useful preliminary screening tool before administering a gold standard test, which generally requires competent physician analyses. In essence stating that an abnormal ECG read on your wearable could warrant a trip to the doctor for further testing that may or may not reveal sinister pathology.

“Never before have we known how cost-effective and efficient it can be to monitor patients in remote settings than in the past year”

When I asked a few cardiologists to give me their opinion on wearables, they all seemed to unanimously agree that these devices may inaccurately pick up signals that are normal or miss signals that are abnormal, essentially making it evident that more robust studies in non-stationary conditions are needed with appropriate methodology in terms of number of subjects involved, acquisition and analysis, techniques implied, before being able to recommend any of the commercially available devices.

An iterative application of a framework consisting of robust validation protocols can create a huge plethora of datasets from these wearables that could change the way we monitor and rehabilitate our patients. Never before have we known how cost-effective and efficient it can be to monitor patients in remote settings than in the past year.

In the world of physical therapy, remote patient monitoring using wearables can bring us a certain distance in knowing our patients even before we physically see them, which when combined with our expertise can provide us with useful lifestyle information that could otherwise be overlooked or assumed. Remote vital-signs monitoring to assess medication compliance and prevent strokes, step-counter to measure physical activity, gait speed and cadence to prevent falls, jolt-sensor head bands to detect concussions, exoskeletons to assist with movement impairments, app-based home exercise programs - are simple but effective measures of gathering preliminary information about our patients and in some cases can prevent comorbidities and chronic disease.

Many systematic reviews studying the validity of wearable technology acknowledged that whilst there are drawbacks in how the data gathered should be interpreted; wearable technology shows increased motivation, self-determination and social connectedness. Interesting to note here is that many studies found a variable acceptance rate among age groups with younger adults/adolescents and physically active individuals at baseline not showing much benefit from such technology. However, patients that had an incentive or goal demonstrated more engagement, compliance, and carryover of good lifestyle habits.

It appears that health care could be diametrically different once difficulties in achieving user-friendly solutions, security and privacy concerns, industry standards, and various technical bottlenecks are overcome. Until then, I would suggest interpreting the data on your wearable with a pinch of salt, or in scientific jargon with a 5-10 percent mean percentage error.