Machine Learning System Uses Smart Speakers to Monitor Heartbeat

Using smart speakers and machine learning algorithms, researchers from University of Washington (UW) have developed a way to monitor both regular and irregular heartbeats without physical contact.

In a study published in Communications Biology, the team described how the system sends inaudible sounds from the speaker out into a room and, based on the way the sounds are reflected back to the speaker, it can identify and monitor individual heartbeats.

Because the heartbeat is such a small motion on the chest surface, the system uses machine learning to help the smart speaker locate both regular and irregular heartbeats.

Researchers noted that while most people are familiar with the concept of a heart rate, providers are more interested in assessing heart rhythm. Heart rate is the average number of heartbeats over time, while a heart rhythm describes the pattern of heartbeats.

For example, if a person has a heart rate of 60 beats per minute, they could have a regular heart rhythm of one beat every second, or they could have an irregular heart rhythm, where beats are randomly scattered across that minute but still average out to 60 beats per minute.

“Heart rhythm disorders are actually more common than some other well-known heart conditions. Cardiac arrhythmias can cause major morbidities such as strokes, but can be highly unpredictable in occurrence, and thus difficult to diagnose,” said co-senior author Dr. Arun Sridhar, assistant professor of cardiology at the UW School of Medicine.

“Availability of a low-cost test that can be performed frequently and at the convenience of home can be a game-changer for certain patients in terms of early diagnosis and management.”

The key to evaluating heart rhythm is identifying the individual heartbeats, researchers noted. For this system, the search for heartbeats begins when a person sits within one to two feet of in front of a smart speaker. Then the system plays an inaudible continuous sound that bounces off the person and back to the smart speaker.

Based on how the returned sound has changed, the system can isolate movements on the person, including the rise and fall of their chest as they breathe.

“The motion from someone’s breathing is orders of magnitude larger on the chest wall than the motion from heartbeats, so that poses a pretty big challenge,” said lead author Anran Wang, a doctoral student in the Allen School.

“And the breathing signal is not regular so it’s hard to simply filter it out. Using the fact that smart speakers have multiple microphones, we designed a new beam-forming algorithm to help the speakers find heartbeats.”

For the system, the team designed a supervised machine learning algorithm that learns on the fly rather than from a training set. This algorithm combines signals from all of the smart speaker’s multiple microphones to identify the heartbeat signal.

“This is similar to how Alexa can always find my voice even if I’m playing a video or if there are multiple people talking in the room,” said co-senior author Shyam Gollakota, a UW associate professor in the Paul G. Allen School of Computer Science & Engineering.

“When I say, ‘Hey, Alexa,’ the microphones are working together to find me in the room and listen to what I say next. That’s basically what’s happening here but with the heartbeat.”

The heartbeat signals that the smart speaker detects don’t look like the usual peaks commonly associated with traditional heartbeat monitors. Researchers used a second algorithm to segment the signal into individual heartbeats so that the system could extract the inter-beat interval, or the amount of time between two heartbeats.

“With this method, we are not getting the electric signal of the heart contracting. Instead, we’re seeing the vibrations on the skin when the heart beats,” Wang said.

In the study, researchers tested a prototype smart speaker on two groups: 26 healthy participants and 24 hospitalized patients with a range of cardiac conditions, including atrial fibrillation and heart failure.

The team compared the smart speaker’s inter-beat interval with one from a standard heartbeat monitor. Of the nearly 12,300 heartbeats measured for the healthy participants, the smart speaker’s median inter-beat interval was within 28 milliseconds of the standard monitor.

The results also showed that the smart speaker performed almost as well with cardiac patients. Of the more than 5,600 heartbeats measured, the median inter-beat interval was within 30 milliseconds of the standard.

“Regular heartbeats are easy enough to detect even if the signal is small, because you can look for a periodic pattern in the data,” said Gollakota.

“But irregular heartbeats are really challenging because there is no such pattern. I wasn’t sure that it would be possible to detect them, so I was pleasantly surprised that our algorithms could identify irregular heartbeats during tests with cardiac patients.”

The system is currently set up for spot checks, meaning if a person is concerned about their heart rhythm, they can sit in front of a smart speaker to get a reading. In future versions of the system, the researchers believe that the tool could continuously monitor heartbeats while people are asleep, which could help providers diagnose conditions like sleep apnea.

“If you have a device like this, you can monitor a patient on an extended basis and define patterns that are individualized for the patient. For example, we can figure out when arrhythmias are happening for each specific patient and then develop corresponding care plans that are tailored for when the patients actually need them,” Sridhar said.

“This is the future of cardiology. And the beauty of using these kinds of devices is that they are already in people’s homes.”