Artificial Intelligence Translates Mental Handwriting into Text

Artificial intelligence software and a brain-computer interface (BCI) device have enabled a man with full-body paralysis to communicate by text at a rate of 18 words per minute, according to a study conducted by Stanford University researchers, published in Nature.  

“We’ve learned that the brain retains its ability to prescribe fine movements a full decade after the body has lost its ability to execute those movements,” said lead author and research scientist at the Howard Hughes Medical Institute, Frank Willett, PhD, in a May 12 press release.  

“And we’ve learned that complicated intended motions involving changing speeds and curved trajectories, like handwriting, can be interpreted more easily and more rapidly by the artificial-intelligence algorithms we’re using than can simpler intended motions like moving a cursor in a straight path at a steady speed. Alphabetical letters are different from one another, so they’re easier to tell apart.” 

For context, able-bodied people can type approximately 23 words per minute on a smartphone, the study reported. 

Two BCI chips were placed on the left side of the participant’s (referred to as T5) brain, picking up neural signals firing in a specific part of the motor cortex that dictates hand movement. The artificial intelligence program interprets the signals and then displays the subject’s hand motions on a computer screen.   

In a 2017 study by the same researchers, the same participant was able to produce approximately 40 characters per minute. Researchers in the 2017 study asked participants to focus on using an imaginary computer mouse and then click on a keyboard display, while the 2021 study asked participants to hand-write letters on an imaginary piece of paper.  

In repeating each letter of the alphabet 10 times, the software was able to learn to associate the participant’s effort to write particular letters with corresponding neural signals.   

“In further sessions, T5 was instructed to copy sentences the algorithms had never been exposed to. He was eventually able to generate 90 characters, or about 18 words, per minute,” the press release stated. 

In later sessions, T5 was asked to copy sentences that were unfamiliar to the algorithm and was eventually able to reach an average of about 18 words per minute. When answering open-ended questions, T5 produced about 15 words per minute on average, according to the study.  

T5’s results showed approximately one mistake every 18 or 19 characters when copying sentences, and one in every 11 or 12 characters in free-composition. However, the program achieved 99 percent accuracy with the addition of autocorrect. The participant tripled the previous free-composition record, also set by him in 2017.  

Currently, the BCI technology developed for this study is not approved for commercial use, but Stanford University’s Office of Technology Licensing recently applied for an intellectual property patent. 

The study’s findings “could spur further advances benefiting hundreds of thousands of Americans, and millions globally, who’ve lost the use of their upper limbs or their ability to speak due to spinal-cord injuries, strokes or amyotrophic lateral sclerosis, also known as Lou Gehrig’s disease,” suggested professor of neurosurgery Jaimie Henderson, MD, in the press release.  

From fueling a rapid COVID-19 antibody test to helping robots navigate the ED, developments in artificial intelligence have shown potential to propel scientific advancement and lead to better health outcomes in the future.