Brain activity decoders, also known as brain-computer interfaces (BCIs), are cutting-edge technologies capable of interpreting and decoding brain activity into meaningful data. While brain decoding is still in its infancy, researchers have made significant progress in decoding certain aspects of brain activity, such as simple visual stimuli or basic thoughts.
A semantic decoder, a new artificial intelligence system, can convert a person’s brain activity while listening to a story or silently imagining telling a story into a continuous stream of text. Researchers at The University of Texas at Austin developed a system that could help people who are mentally conscious but unable to physically speak, such as stroke victims, communicate intelligibly again.
Jerry Tang, a doctoral student in computer science, and Alex Huth, an assistant professor of neuroscience and computer science at UT Austin, led the study, which was published in the journal Nature Neuroscience. The work is partly based on a transformer model, similar to those used by Open AI’s ChatGPT and Google’s Bard.
Unlike other language decoding systems under development, this one does not require surgical implants on subjects, making the process noninvasive. Participants are also not required to use only words from a predetermined list. After extensive training of the decoder, in which the individual listens to hours of podcasts in the scanner, brain activity is measured using an fMRI scanner. Later, provided that the participant is open to having their thoughts decoded, their listening to a new story or imagining telling a story allows the machine to generate corresponding text from brain activity alone.
We take the concerns that it could be used for bad purposes very seriously and have worked hard to avoid that. We want to make sure that people only use these types of technologies when they want to and when it helps them.
Jerry Tang
“For a noninvasive method, this is a real leap forward compared to what’s been done before, which is typically single words or short sentences,” Huth said. “We’re getting the model to decode continuous language for extended periods of time with complicated ideas.”
The end result is not a verbatim transcript. Instead, researchers designed it to capture, albeit imperfectly, the gist of what is being said or thought. When the decoder has been trained to monitor a participant’s brain activity, the machine produces text that closely (and sometimes precisely) matches the intended meanings of the original words about half of the time.
In one experiment, a participant heard a speaker say, “I don’t have my driver’s license yet,” and their thoughts were translated as, “She hasn’t even started learning to drive yet.” Listening to the words, “I couldn’t decide whether to scream, cry, or flee.” Instead, I said, ‘Leave me alone!'” was decoded as, “Started to scream and cry, and then she just said, ‘I told you to leave me alone.'”
Beginning with an earlier version of the paper that appeared as a preprint online, the researchers addressed questions about the potential misuse of the technology. The paper describes how decoding worked only with cooperative participants who had participated willingly in training the decoder. Results for individuals on whom the decoder had not been trained were unintelligible, and if participants on whom the decoder had been trained later put up resistance – for example, by thinking other thoughts – results were similarly unusable.
“We take the concerns that it could be used for bad purposes very seriously and have worked hard to avoid that,” Tang said. “We want to make sure that people only use these types of technologies when they want to and when it helps them.”
The researchers asked subjects to watch four short, silent videos while in the scanner, in addition to listening to or thinking about stories. The semantic decoder was able to accurately describe certain events in the videos by using their brain activity.
Because of the time required on an fMRI machine, the system is currently impractical for use outside of the laboratory. However, the researchers believe that their findings could be applied to other, more portable brain-imaging systems, such as functional near-infrared spectroscopy (fNIRS).
“fNIRS measures where there is more or less blood flow in the brain at different points in time,” Huth explained. “It turns out that this is exactly the same kind of signal that fMRI measures. So, our exact kind of approach should translate to fNIRS,” he said, noting that the resolution with fNIRS would be lower.
