Accuracy of Heart Rate and Respiratory Rate Measurements Using Two Types of Wearable Devices

ABSTRACT

Objectives: Wearable devices such as fitness trackers have become popular in the healthcare field. Tracking heart rate and respiratory rate, in addition to physical activity, may provide an accurate picture of daily health. We believe that a combination of two types of devices can simultaneously measure and record physical activity, heart rate, and respiratory rate. However, the measurement accuracies of these two types of devices are not clear. This study aimed to determine the measurement accuracies of two wearable devices for heart and respiratory rate measurements.

Methods: Ten healthy men performed incremental load tests (ILTs) and constant load tests (CLTs) on a cycle ergometer. The heart and respiratory rates were measured using wrist-worn (Silmee W22, TDK, Japan, Tokyo) and respiratory tracking devices (Spire Stone, Spire Health, San Francisco, CA, USA), respectively. A 12-lead electrocardiograph and the breath-by-breath method were used as external standards for heart and respiratory rates, respectively.

Results: Bland–Altman analysis showed that heart rate had a fixed bias at rest and during ILT and CLT and had a proportional bias during CLT. The standard error values of the regression at rest and during CLT were less than 10 bpm for heart rate and less than 5.0 /min for respiratory rate. During ILT, the standard error was greater than 10 bpm for heart rate and approximately 5.0 /min for respiratory rate.

Conclusions: The heart and respiratory rate measurements obtained using wearable devices were accurate within the practical margin of error.

INTRODUCTION

Since 2014, the use of wearable devices such as smart watches has become increasingly common in the healthcare field.¹⁾ In particular, many reports have discussed fitness trackers that measure physical activity with a built-in accelerometer,^2,3) and levels of reliability have been reported.⁴⁾ By assessing changes in heart rate and respiratory rate, these devices can provide an accurate picture of overall health in daily life, especially when used with other information regarding physical activity.

Most wrist-worn devices measure heart rate through the principle of photoplethysmography (PPG). PPG measures the heart rate by sensing the blood flow in capillaries based on the amount of reflected green light from a light-emitting diode (LED) reaching a photodiode. However, PPG may also recognize and record large continuous body movements (more intense than walking) as heart rates because of their similar frequencies. Moreover, the fitness and skin color of the wearer can also affect the measurement. Conventional wrist-worn heart rate sensors are designed to display estimated values with compensation for errors caused by body movement, but the detection of heart rate abnormalities, such as tachycardia and extra-phase heartbeats, is difficult.

To date, the only device reported to be capable of accurate measurement of respiratory rate is a sensor worn on the chest.^5,6) Tight-fitting smart shirts, which are equipped with sensors, can simultaneously measure physical activity, heart rate, and respiratory rate. These functional garments reportedly have high measurement accuracy during various forms of exercise.^7,8) However, they are not suitable for everyday repetitive use because a gel is needed to adhere the electrocardiograph (ECG) sensor to the skin. In addition, a study that examined the usability of wearable devices in older Irish people found that shirt-type sensors did not perform as well as other devices in terms of usability, acceptability, and motivation.⁹⁾ To improve compliance, the study recommended a watch-like device that can be worn easily.⁹⁾

We believe that combined use of two types of devices, which are easy to use repeatedly in daily life, can simultaneously record physical activity, heart rate, and respiratory rate. However, the measurement accuracies of these devices are not clear. The purpose of this study was to compare the accuracy of heart rate and respiratory rate data recorded by different wearable devices with measurements recorded by standard methods.

MATERIALS AND METHODS

Participants

Ten healthy men [age 26 ± 3 years; height, 171.3 ± 3.2 cm; weight, 62.1 ± 7.2 kg; body mass index, 21.1 ± 2.2 kg/m² (means ± SDs)] participated in this study. Grounds for exclusion from the study were age younger than 20 years, underlying disease, or the inability to complete the exercise tests.

Ethical Considerations

This study was conducted with the approval of the Ethics Committee of Akita University Hospital (approval number 2498). In accordance with the Declaration of Helsinki, the subjects were fully informed about the purpose of the research and their voluntary written consent was obtained regarding participation in the study and the handling of their research-related personal information.

Wearable Devices

Heart Rate Monitoring Device

The heart rate was measured with a wrist-worn device (Silmee W22, TDK, Tokyo, Japan) that measured physical activity¹⁰⁾ and heart rate through PPG. To obtain more accurate data, this device was designed to use estimated heart rate values in such a way that in the case of large body movements, wrong values were not recorded. In addition, the device included a function to automatically adjust the amount of LED light emission because heart rate measurement with LED sensors may vary according to individual differences in blood flow and skin temperature.

Respiratory Rate Monitoring Device

The respiratory rate was measured with a respiratory tracking device (Spire Stone, Spire Health, San Francisco, CA, USA) that used a built-in pressure sensor to monitor the depth of respiration based on the expansion and contraction of the abdomen or chest. When the device detected a large body movement, it switched to activity mode and counted the number of steps taken instead of the respiratory rate. The device was also capable of converting the respiratory patterns and activity into processed data by a proprietary algorithm to determine when a person was breathing slower, faster, more irregularly, or more consistently than usual, based on the consistency of the “baseline” respiratory rate. The respiratory patterns were classified according to speed and regularity as: calm (slow and smooth respiratory rhythm), focus (medium and stable respiratory rhythm), and tense (fast and unstable respiratory rhythm). Furthermore, the waveform and respiratory rate were displayed in real time on a smartphone connected via Bluetooth.

Heart Rate and Respiratory Rate Monitors

The heart rate was measured every 60 s with a wrist-worn device. The measured values were recorded in the device’s internal memory and output was generated using the designated analysis software (Silmee ProWx, TDK, Tokyo, Japan). As an external standard, a 12-lead ECG sensor (FX-7542, Fukuda Denshi, Tokyo, Japan) was used to measure heart rate.

The respiratory rate was recorded every 30 s using the designated respiratory device, with real-time readings displayed on a smartphone. As an external standard, the breath-by-breath method was applied to measure respiratory rate using an exhaled gas analyzer (Aeromonitor AE-310, Minato Medical Science, Tokyo, Japan).

Exercise Protocol

Each participant wore the two wearable devices, and heart rate and respiratory rate were measured during exercise tests performed on a cycle ergometer. Two types of exercise tests were conducted: the incremental load test (ILT) and the constant load test (CLT).

The ILT began with 4 min of rest and a 4-min 20-W warm-up, followed by a continuous 2-W increase in the work load every 6 s. Participants were instructed to maintain cadences at 60 rpm. The ILT was terminated when the heart rate reached 85% of the maximum heart rate (THR), systolic blood pressure was greater than 220 mmHg or progressively decreased, oxygen saturation was less than 90%, cadences were reduced, or when other subjective symptoms were noted.

The intensity of the CLT was calculated from the maximum intensity of motion (peak load) obtained during ILT. The CLT started with 3 min of rest, followed by 6 min of cycling at 30% of the peak load, and then the load was increased to 60% of the peak load for 6 min (12 min in total). To account for the delay in the respiratory circulation response to exercise, we analyzed the data obtained in the last 3 min at both intensities.

Statistical Analysis

Data were used to calculate the mean ± SD for continuous variables. The presence of systematic errors was investigated using Bland–Altman analysis, and the limits of agreement (LOAs) were calculated to clarify the range of practical errors. In addition, single regression analysis was performed with measurements obtained from the wearable devices as the independent variables and external criteria (heart rate measured by ECG and respiratory rate measured by the breath-by-breath method) were used as the dependent variables. First-order regression lines were calculated for measurements obtained at rest and during ILT and CLT. R² and the mean absolute error (MAE) were calculated using the grouped tenfold cross-validation method to test model fitting. R version 3.6.1 (R Foundation for Statistical Computing) and RStudio version 1.2.5042 (RStudio, Boston, MA, USA) were used for statistical analysis. The level of significance was set at P <0.05.

RESULTS

All participants completed both the ILT and ILR exercise tests. The reasons for termination of the ILT were decreased cadence in seven subjects and 85% THR attained in three subjects. Data relating to a sudden drop in the heart rate (<80 bpm measured with a wrist-worn device or a difference between measured heart rates >20 bpm) during ILT were excluded from the analysis as measurement errors (Fig. 1). In such cases, it was considered that the wrist-worn device measured pedal cadence rather than heart rate, and visual observation was able to confirm such measurements as incorrect.

Fig. 1.

Correlation between heart rate measured with wrist-worn device and that measured with electrocardiogram (ECG) during incremental load test. Sudden drops in the heart rate (<80 bpm measured with a wrist-worn device or a difference between measured heart rates >20 bpm) during ILT were excluded from the analysis as measurement errors because the wrist-worn device likely detected the pedal cadence. Visual observation clearly determined that such measurements were inappropriate. bpm, beats per minute.

Systematic Bias

The mean differences in heart rate between the wrist-worn device and ECG were −1.6 ± 4.2 bpm during ILT, 6.1 ± 9.9 bpm during CLT, and −0.8 ± 6.3 bpm at rest. The mean differences in respiratory rate between a wrist-worn device and the breath-by-breath method were 0.5 ± 2.0 /min during ILT, −1.6 ± 2.2 /min during CLT, and −0.9 ± 2.3 /min at rest.

The results of the Bland–Altman analysis are given in Table 1, and the data were used to construct Figs. 2–4. The heart rate showed fixed bias at rest as well as during ILT and CLT and showed proportional bias during CLT. At rest and during ILT, there was no proportional bias in heart rate. The respiratory rate had fixed bias during ILT and CLT. Moreover, proportional bias was observed at rest and during ILT and CLT. There was no fixed bias at rest for respiratory rate. The limit of agreement (LOA) of heart rate at rest was within ±10.0 bpm, whereas it was outside the range of ±10.0 bpm during ILT and CLT. The LOA of respiratory rate was within ±5.0 /min at rest, whereas it ranged from −6.24 to 2.69 /min during ILT and from −5.48 to 3.71 /min during CLT.

Table 1. Summary of the results of Bland–Altman analysis

		Fixed bias			Proportional bias			LOA
		95% CI	Y/N		r	P value	Y/N
Wrist-worn device heart rate	Rest	−3.19, −0.07	Y		−0.055	0.780	N	−9.79, 6.53
	ILT	−17.66, −7.12	Y		−0.152	0.420	N	−38.31, 12.93
	CLT	−7.04, −2.19	Y		0.382	0.004	Y	−22.22, 12.98
Respiratory tracking device respiratory rate	Rest	−0.18, 1.24	N		−0.353	0.030	Y	−3.79, 4.83
	ILT	−2.10, −1.45	Y		−0.278	<0.001	Y	−6.24, 2.69
	CLT	−1.49, −0.28	Y		−0.279	0.030	Y	−5.48, 3.71

CI, confidence interval; Y, yes; N, no.

Fig. 2.

Bland–Altman plots and 95% limits of agreement for at-rest measurements of heart rate (HR) (left) and respiratory rate (right).

Fig. 3.

Bland–Altman plots and 95% limits of agreement during incremental load tests for measurement of heart rate (HR) (left) and respiratory rate (right).

Fig. 4.

Bland–Altman plots and 95% limits of agreement during constant load tests for measurement of heart rate (HR) (left) and respiratory rate (right).

Regression Analysis

Single regression analysis with heart rate measured by ECG as the dependent variable and heart rate measured by the wrist-worn device as the independent variable yielded significant regression equations at rest and during ILT and CLT. The standard errors of the estimates were less than 10.0 bpm at rest and during CLT, and more than 10.0 bpm during ILT (Table 2). Tenfold cross-validation analysis gave a root mean square error (RMSE) of 8.380189, R² of 0.88896, and an MAE of 6.298022.

Table 2. Summary of the results of regression analysis

		R	R2	Adjusted R2	SEE	P value	95% CI
Wrist-worn device
	Rest	0.939	0.881	0.877	4.20611	<0.001	0.820, 1.093
	ILT	0.877	0.768	0.760	13.23761	<0.001	0.740, 1.150
	CLT	0.942	0.887	0.885	7.72141	<0.001	0.740, 0.901
Respiratory tracking device
	Rest	0.691	0.477	0.458	3.69535	<0.001	0.498, 1.178
	ILT	0.679	0.461	0.456	5.24037	<0.001	0.667, 1.034
	CLT	0.803	0.644	0.640	2.64804	<0.001	0.815, 1.128

SEE, standard error of estimates.

The following regression equations for heart rate were obtained:

At rest:

ECG heart rate=4.412 + 0.956 × wrist-worn device heart rate (R²=0.881, P <0.001)

During ILT:

ECG heart rate=18.669 + 0.945 × wrist-worn device heart rate (R²=0.768, P <0.001)

During CLT:

ECG heart rate =21.840 + 0.820 × wrist-worn device heart rate (R²=0.887, P <0.001)

Single regression analysis with respiratory rate measured by the breath-by-breath method as the dependent variable and respiratory rate measured by the respiratory tracking device as the independent variable yielded significant regression equations at rest and during ILT and CLT. The standard errors of the estimates were less than 5.0 /min at rest and during CLT, and approximately 5.0 /min during ILT (Table 2). Tenfold cross-validation analysis gave an RMSE of 5.978712, R² of 0.6980266, and an MAE of 2.514456.

The following regression equations for respiratory rate were obtained:

At rest:

Breath-by-breath respiratory rate=2.372 + 0.838 × respiratory tracking device respiratory rate (R²=0.477, P <0.001)

During ILT:

Breath-by-breath respiratory rate=3.700 + 0.850 × respiratory tracking device respiratory rate (R²=0.461, P <0.001)

During CLT:

Breath-by-breath respiratory rate=1.279 + 0.971 × respiratory tracking device respiratory rate (R²=0.644, P <0.001)

DISCUSSION

In this study, we examined the accuracy of a wrist-worn device and a respiratory tracking device by correlating their data with measurements recorded using standard procedures. Both devices showed high correlation with external criteria (wrist-worn device: r=0.877–0.942, respiratory tracking device: r=0.679–0.803). The errors at rest were small (wrist-worn device: <10.0 bpm, respiratory tracking device: <5.0 /min) and relatively large during exercise. However, by using regression equations, we were able to keep the errors within practical limits.

Accuracy of Wearable Devices

Accuracy of Wrist-worn Device

In previous studies that compared the accuracy of heart rate measurement using wearable devices with heart rate measured by ECG, the chest band sensor (worn as a band on the chest) was found to be the most accurate. All studies reported an accuracy of r=0.99, which makes the measurements of chest band sensors almost identical to that of the ECG.^11,12,13) Among the wrist-worn devices, the highest correlation was r=0.91–0.92.^12,13) Other devices have been reported to have r=0.67‒0.91.^12,13) The wrist-worn device validated in this study was found to have a high correlation coefficient (r=0.88–0.94).

All previous studies have reported the existence of an error between the ECG and the wrist-worn device data. The LOAs of devices with correlation coefficients of approximately 0.90 are reportedly −27 to +29 bpm and −17 to +20 bpm during multi-step exercise tests performed on a treadmill and cycle ergometer.^11,12,13) In the current study, the LOA of the wrist-worn device during ILT (−38.31 to +12.93 bpm) was greater than that of devices with minimal errors, but the LOA during CLT (−22.22 to +12.98 bpm) was similar to that of devices with correlation coefficients of approximately 0.90. The LOAs for devices with correlation coefficients of approximately 0.80 were reported to be −24 to +31 bpm for low values¹³⁾ and −34 to +39 bpm for high values.¹²⁾ In this context, the accuracy of the wrist-worn device in the current study is similar to or better than that of other PPG sensors.

Notably, measurement errors constituted a high percentage (37.4%) of the total data. This may have been caused by the tendency of the participants to grip the handle of the cycle ergometer more tightly at increased load, which in turn may have affected body movement. In particular, sudden drops in the heart rate measured with the wrist-worn device to around 60 bpm (while the ECG heart rate gradually increased) are likely to reflect the pedal cadence (60 rpm). To address these measurement errors, we propose the use of an algorithm to improve the accuracy of measurements during exercise by combining data from the built-in acceleration sensor in the wrist-worn device.

Accuracy of Respiratory Tracking Device

There are few reports on the use of wearable devices to measure respiratory rate.¹⁴⁾ The correlation between the respiratory rates measured with a small body sensor that attaches to the patient’s chest and those obtained by visual measurement was reported to be as low as r=0.39.⁵⁾ Smith et al.⁷⁾ examined the correlation between the respiratory rates measured with a shirt-type sensor and with the breath-by-breath method for ILTs performed on a cycle ergometer. They found that the correlation coefficients were highest during submaximal exercise (r=0.95), followed by rest (r=0.88) and maximal exercise (r=0.84) in that order.⁷⁾ Furthermore, the mean difference between values obtained by the two methods at rest and during submaximal exercise was 1.0 ± 3.0 /min, whereas that at maximal exercise was 2.0 ± 7.0 /min. In the present study, the correlation coefficient between respiratory rates measured with the respiratory tracking device and with the breath-by-breath method was 0.68–0.80, and the mean differences (resting, 0.5 ± 2.0 /min; ILT, ‒1.6 ± 2.2 /min; CLT, ‒0.9 ± 2.3 /min) were similar to those for shirt-type sensors. However, proportional and fixed biases were observed, and they tended to underestimate the results during exercise.

The parameters for assessing respiration are the number of breaths, the depth of breathing (shallow or deep), and the breathing pattern (chest or abdominal). The breath-by-breath method measures exhaled gas directly, so the influences of breath depth and pattern on the measurement are minimal. The shirt-type sensor has two bands of sensors—one at the top of the chest wall and the other at the bottom—so the effects of respiration depth and pattern are considered to be small. In contrast, the respiratory tracking device has only one sensing range, which we considered to contribute toward underestimates because the device cannot track shallow breathing or changes in respiration patterns. However, using the regression equation, the margin of error in this study was within ± 5.0 /min, which we considered to be within the acceptable range of error. Considering the small size of the sensing area, we conclude that the respiratory tracking device has reasonable accuracy.

Usefulness in Daily Living

Activities of daily living include sitting and standing activities such as going to the toilet, reading, eating, washing dishes, washing clothes, and watching television. These activities are similar to CLT because they involve relatively few body movements and constant motor intensity. In addition, in situations where cycle ergometers are used for exercise (e.g., rehabilitation and fitness), the accuracy of the measurements recorded by wearable devices would be similar to that during CLT. Because wearable devices have high measurement accuracy at rest and during CLT, they can be expected to accurately measure heart rate and respiratory rate in daily life.

Concerning the long-term use of the devices, wrist-worn devices that also function as watches are reported to be more versatile and easier to use than others.⁹⁾ Although chest band sensors and shirt-type sensors provide better measurement accuracy than wrist-worn devices, their chest bands and tight shirts are not suitable for daily use. There is also a user engagement issue, which makes it only compatible with certain operating systems. To its credit, the respiratory tracking device evaluated in this study is small, easy to wear, and suitable for daily use. Furthermore, it has the advantage that the respiratory waveform can be checked in real time to confirm the appropriateness of the measurement. In particular, when used in the healthcare field, the wearable devices evaluated in this study may be more acceptable to the elderly and physically challenged than other more complicated devices.

Study Limitations

This study has several limitations. First, it was conducted in a stable experimental environment, and its applicability to real-life situations needs to be examined in the future. Second, the wrist-worn device needs an analysis algorithm to accurately detect measurement errors. However, the ability of the sensor to reflect the pedal cadence on a stationary cycle ergometer (Supplementary Material) may also be necessary in some situations. Therefore, the user can assume that measurement errors may occur. Finally, the participants in this study were all men. Therefore, we were not able to examine the effects of gender differences, such as chest circumference and respiratory patterns, on the study results. However, chest circumference is not expected to have a significant effect on the measurement because the sensors are not worn over the entire chest area.

CONCLUSION

This study found that heart rate measurement with a wrist-worn device was highly accurate and that the regression equation developed in this study was able to maintain accuracy within a practical margin of error. However, in the case of ILT, the results were outside the practical error range, even after adjusting for the errors. The respiratory rate measured with the respiratory tracking device showed high correlation with the breath-by-breath method (although there were fixed and proportional biases), and the changes in respiration could be adequately tracked. Both wearable devices used in this study can detect body movement and can disqualify measurements by identifying artefacts, thereby allowing accurate data to be accumulated in daily life activities.

ACKNOWLEDGMENTS

The authors thank all those who participated in the study. This work was supported by a Grant-in-Aid for Scientific Research from the Japan Society for the Promotion of Science (grant number 19K197887).

CONFLICTS OF INTEREST

The authors report no conflicts of interest.

REFERENCES

• 1.  Ramkumar PN, Haeberle HS, Ramanathan D, Cantrell WA, Navarro SM, Mont MA, Bloomfield M, Patterson BM: Remote patient monitoring using mobile health for total knee arthroplasty: validation of a wearable and machine learning-based surveillance platform. J Arthroplasty 2019;34:2253–2259. PMID:31128890, DOI:10.1016/j.arth.2019.05.021
• 2.  Wile DJ, Ranawaya R, Kiss ZH: Smart watch accelerometry for analysis and diagnosis of tremor. J Neurosci Methods 2014;230:1–4. PMID:24769376, DOI:10.1016/j.jneumeth.2014.04.021
• 3.  Kwon SB, Ahn JW, Lee SM, Lee J, Lee D, Hong J, Kim HC, Yoon HJ: Estimating maximal oxygen uptake from daily activity data measured by a watch-type fitness tracker: cross-sectional study. JMIR Mhealth Uhealth 2019;7:e13327. PMID:31199336, DOI:10.2196/13327
• 4.  Burton E, Hill KD, Lautenschlager NT, Thøgersen-Ntoumani C, Lewin G, Boyle E, Howie E: Reliability and validity of two fitness tracker devices in the laboratory and home environment for older community-dwelling people. BMC Geriatr 2018;18:103. PMID:29724191, DOI:10.1186/s12877-018-0793-4
• 5.Keogh A, Dorn JF, Walsh L, Calvo F, Caulfield B: Comparing the usability and acceptability of wearable sensors among older Irish adults in a real-world context: observational study. JMIR Mhealth Uhealth 2020;8:e15704.
• 6.  Izmailova ES, McLean IL, Hather G, Merberg D, Homsy J, Cantor M, Volfson D, Bhatia G, Perakslis ED, Benko C, Wagner JA: Continuous monitoring using a wearable device detects activity-induced heart rate changes after administration of amphetamine. Clin Transl Sci 2019;12:677–686. PMID:31365190, DOI:10.1111/cts.12673
• 7.  Smith CM, Chillrud SN, Jack DW, Kinney P, Yang Q, Layton AM: Laboratory validation of hexoskin biometric shirt at rest, submaximal exercise, and maximal exercise while riding a stationary bicycle. J Occup Environ Med 2019;61:e104–e111. PMID:30946696, DOI:10.1097/JOM.0000000000001537
• 8.  Elliot CA, Hamlin MJ, Lizamore CA: Validity and reliability of the hexoskin wearable biometric vest during maximal aerobic power testing in elite cyclists. J Strength Cond Res 2019;33:1437–1444. PMID:28759538, DOI:10.1519/JSC.0000000000002005
• 9.  Villar R, Beltrame T, Hughson RL: Validation of the hexoskin wearable vest during lying, sitting, standing, and walking activities. Appl Physiol Nutr Metab 2015;40:1019–1024. PMID:26360814, DOI:10.1139/apnm-2015-0140
• 10.  Kawajiri M, Nakamura Y, Takeishi Y, Ito N, Atogami F, Yoshizawa T: Longitudinal study of physical activity using an accelerometer in Japanese pregnant women. Jpn J Nurs Sci 2020;17:e12294. PMID:31465152, DOI:10.1111/jjns.12294
• 11.  Wallén MB, Hasson D, Theorell T, Canlon B, Osika W: Possibilities and limitations of the polar RS800 in measuring heart rate variability at rest. Eur J Appl Physiol 2012;112:1153–1165. PMID:21766225, DOI:10.1007/s00421-011-2079-9
• 12.  Wang R, Blackburn G, Desai M, Phelan D, Gillinov L, Houghtaling P, Gillinov M: Accuracy of wrist-worn heart rate monitors. JAMA Cardiol 2017;2:104–106. PMID:27732703, DOI:10.1001/jamacardio.2016.3340
• 13.  Gillinov S, Etiwy M, Wang R, Blackburn G, Phelan D, Gillinov AM, Houghtaling P, Javadikasgari H, Desai M: Variable accuracy of wearable heart rate monitors during aerobic exercise. Med Sci Sports Exerc 2017;49:1697–1703. PMID:28709155, DOI:10.1249/MSS.0000000000001284
• 14.  Alharbi M, Straiton N, Smith S, Neubeck L, Gallagher R: Data management and wearables in older adults: a systematic review. Maturitas 2019;124:100–110. PMID:30910279, DOI:10.1016/j.maturitas.2019.03.012