Machine learning framework captures uncertainty in medical images

Researchers from the Massachusetts Institute of Technology (MIT), Massachusetts General Hospital and the Broad Institute of MIT and Harvard have developed a machine learning approach to help capture the uncertainty present in medical image segmentation, which could improve clinical decision-making.

Medical image segmentation is a critical aspect of image analysis, as it allows clinicians to extract a region of interest – such as tissues or lesions – from the image. To do so, clinicians annotate pixels of interest, which are then used to divide the image into labelled regions.

The process is often deployed in clinical quantification, diagnosis and surgical planning, and artificial intelligence (AI) has shown promise in assisting users by highlighting pixels of interest for further analysis.

However, AI-driven models usually provide only one output, which limits their utility due to the range of possible segmentations clinicians can provide for a given image. The research team indicated that often, multiple expert human annotators can provide multiple different segmentations for the same image due to differences in interpretation.

These differences represent uncertainty that is inherent to current medical image segmentation approaches, but the researchers underscored that such uncertainty is actually valuable.

“Having options can help in decision-making. Even just seeing that there is uncertainty in a medical image can influence someone’s decisions, so it is important to take this uncertainty into account,” explained lead study author Marianne Rakic, an MIT computer science PhD candidate, in the news release.

By only producing one output, the research team noted, current AI models cannot accurately capture uncertainty.

“Ambiguity has been understudied. If your model completely misses a nodule that three experts say is there and two experts say is not, that is probably something you should pay attention to,” said senior author Adrian Dalca, PhD, an assistant professor at Harvard Medical School and MGH, and a research scientist in the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL).

Further, to perform a new type of segmentation task, these tools would require retraining or fine-tuning, which requires machine learning expertise and significant resource investment that many researchers don’t have.

To address these shortcomings, the research team built Tyche, a machine learning framework that can capture uncertainty in medical images and generate multiple outputs while not requiring retraining for new tasks.

Image segmentation models typically rely on neural networks – AI that uses interconnected layers of nodes and neurons, similar to the human brain – to process data and generate desired outputs. The major shortcomings of medical segmentation models are linked to this neural network infrastructure; the way data are filtered through the model's layers contributes to its inability to capture uncertainty or adapt to new imaging tasks.

Methods to overcome one of these pitfalls exist, but addressing both remains a significant challenge.

“If you want to take ambiguity into account, you often have to use an extremely complicated model. With the method we propose, our goal is to make it easy to use with a relatively small model so that it can make predictions quickly,” Rakic stated.

To achieve this, the researchers designed Tyche to utilize a modified neural network architecture that takes advantage of a ‘context set’ of images. As few as 16 images showing the segmentation task can be fed to the model, which it then uses to generate segmentation predictions.

The context set enables the model to make good predictions without requiring retraining, and there is no limit to the number of ‘context set’ images researchers can use.

This framework also allows Tyche to capture uncertainty. The modified neural network architecture was designed so that as data pass through each layer, potential segmentations produced can ‘talk’ to one another and to the context images to enhance predictions.

Thus, the model can generate as many candidate segmentations as a user desires to capture multiple different, but plausible examples of uncertainty.

“It is like rolling dice. If your model can roll a two, three, or four, but doesn’t know you have a two and a four already, then either one might appear again,” Rakic said.

The tool’s architecture also ensures that it is rewarded for maximizing the quality of its predictions, which the research team asserted gives the model an edge over other approaches.

When tested on sets of annotated medical images, Tyche quickly generated high-quality predictions that captured the annotation diversity of human experts.

Moving forward, the research team aims to expand the tool’s capabilities by incorporating a more flexible ‘context set’ comprised of various image types and text.