Machine Learning Can Predict Heart Failure from a Single X-Ray

Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have developed a machine learning tool that can detect the severity of excess fluid in patients’ lungs – a common warning sign of acute heart failure.

The team noted that each year, about one out of eight deaths in the US is caused at least in part by heart failure. Excess fluid in the lungs, known as pulmonary edema, is a typical early symptom of heart failure, and a patient’s exact level of excess fluid often dictates the doctor’s course of action.

However, making these determinations is difficult, and sometimes requires providers to rely on subtle features in x-rays that can lead to inconsistent diagnoses and treatment plans.

Researchers set out to develop a machine learning model that can quantify the severity of edema on a four-level scale, ranging from 0 (healthy) to 3 (very bad). The team trained the system on more than 300,000 x-ray images, as well as on the corresponding text of reports about the x-rays written by radiologists.

The results showed that the system determined the right level of excess fluid more than half the time, and correctly diagnosed level 3 cases 90 percent of the time. The team was surprised that the system was so successful using the radiologists’ reports, most of which didn’t have labels explaining the exact severity of edema.

“By learning the association between images and their corresponding reports, the method has the potential for a new way of automatic report generation from the detection of image-driven findings,” said Tanveer Syeda-Mahmood, a researcher not involved in the project who serves as chief scientist for IBM’s Medical Sieve Radiology Grand Challenge.

“Of course, further experiments would have to be done for this to be broadly applicable to other findings and their fine-grained descriptors.”

Researchers worked to help the system make sense of the text of the reports, which could often be just a sentence or two. Because different radiologists write with varying tones and use a range of terminology, the team developed a set of linguistic rules and substitutions to ensure that data could be analyzed consistently across reports.

This was in addition to the challenge of designing a model that can jointly train the image and text representations in a meaningful manner.

“Our model can turn both images and text into compact numerical abstractions from which an interpretation can be derived,” said PhD student Geeticka Chauhan. “We trained it to minimize the difference between the representations of the X-ray images and the text of the radiology reports, using the reports to improve the image interpretation.”

The team’s system was also able to explain itself by showing which parts of the reports and areas of x-ray images correspond to the model prediction. Researchers expect that this work will provide more detailed lower-level image-text correlations, so that clinicians can build a taxonomy of images, reports, disease labels, and relevant correlated regions.

“These correlations will be valuable for improving search through a large database of X-ray images and reports, to make retrospective analysis even more effective,” Chauhan said.

The team plans to integrate the platform into Beth Israel Deaconess Medical Center’s emergency room workflow this fall. The model could lead to better edema diagnosis, enabling doctors to manage not only acute heart issues but also conditions like sepsis and kidney failure that are strongly associated with edema.

“This project is meant to augment doctors’ workflow by providing additional information that can be used to inform their diagnoses as well as enable retrospective analyses,” said PhD student Ruizhi Liao.