AI Could Improve Tech Designed to Aid the Visually Impaired

According to researchers from the Pennsylvania State University’s College of Information Sciences and Technology, artificial intelligence (AI) can help solve challenges posed by current computer vision technology, such as remote sighted assistance (RSA).

RSA helps people with low or no vision navigate various tasks that require sight by connecting them with human agents via a video call on their smartphones. However, limitations in existing computer vision technologies mean that they can’t fully support an agent fulfilling certain requests, such as recognizing flight information on an airport’s digital screen or reading instructions on a medicine bottle.

In a study presented at the 27th International Conference on Intelligent User Interfaces in March, the researchers argued that some of these challenges cannot be addressed by existing computer vision technologies alone but warrant new development in human-AI collaboration. John M. Carroll, Ph.D., professor of information sciences and technology, stated that addressing these issues has the potential to advance computer vision technology research and usher in the next generation of RSA service.

“We’re interested in developing this particular paradigm because it is a collaborative activity involving sighted and non-sighted people, as well as computer vision capabilities,” said Carroll in the press release. “We framed it in a very rich way where there are a lot of interesting issues of human-human interaction, human-technology interaction, and technology innovation.”

RSA technology is available through free applications or as a paid service that connects visually impaired users to sighted agents. The live video function of the service allows the agent to look through the user’s camera to assist them with their request.

"For example, creating a worldview by looking through the camera is mentally demanding for the agents,” said Syed Billah, Ph.D., assistant professor of information sciences and technology, in the press release. “The good news is that part of this task can be offloaded to computers running a 3D reconstruction algorithm.”

Such support can be taxing for the agents, especially when assisting users with high-stakes tasks like navigating a parking lot. These are the issues that provide room for improvement in current computer vision technology, Billah said.

Carroll and his collaborators reviewed existing RSA technologies and interviewed users to understand their challenges when using these services. The researchers then identified challenges that could be addressed with existing computer vision technologies and suggested design ideas to address them, such as augmenting video streams with texts, graphics, and detected objects.

But the researchers highlighted five emerging issues that the current technology cannot address because of their complexity. These issues relate to recognizing that objects commonly identified as obstacles by smartphone cameras may not be considered obstacles by visually impaired individuals, helping users navigate their environment when a live camera feed may be lost during low cellular bandwidth, recognizing content on digital LCD displays, recognizing texts on irregular surfaces, and predicting how out-of-frame people or objects will move.

These issues could potentially be addressed by utilizing AI alongside human agents to improve the user experience for visually impaired individuals. However, research and development of such methods are still underway.