How artificial intelligence reads minds

[This is a transcript of the video embedded below.]


Human communication works by turning thought into motion. Whether that’s body language, or speech, or writing – we use muscles in one way or another to get out the information that we want to share. But sometimes it would be really handy if we could communicate directly from our brain, to one another or with a computer. How far is this technology along? How does it work? And what’s next? That’s what we will talk about today.

Scientists currently have two ways to figure out what’s going on inside your brain. One way is to use functional Magnetic Resonance Imaging, the other is using electrodes.

Functional Magnetic Resonance Imaging or fMRI for short measures the flow of blood to different regions of the brain. The blood-flow is correlated with neural activity, so an fMRI tells you what parts of the brain are activated in a certain task. I previously made a video about Magnetic Resonance Imaging, so if you want to know how the physics works, check this out.

The problem with fMRIs is that they require people to lie in a big machine. It isn’t only that using this machine is expensive, it also takes some time to take an fMRI, which means that the temporal resolution isn’t great, typically a few seconds. So fMRI can’t tell you much about fast and temporary processes.

The other way to measure brain activity is electroencephalography, EEG for short, which measures tiny currents in electrodes that are placed on the skin on the head. The advantage of this method is that the temporal resolution is much better. The big disadvantage though is that it gives you only a rough idea about the region where the signal is coming from. A much better way is to put the electrodes directly on the surface of the brain, but this requires surgery.

Elon Musk has the idea that one day people might be willing to have electrodes implanted into their brain and he has put some money behind this with his “neuralink” project. But it’s difficult to get a research project approved if it requires drilling holes into other people’s heads, so most studies currently use fMRI – or people who already have holes in their head for one reason or another.

Before we talk about what recent studies have found, I want to briefly thank our tier four supporters on patreon. Your support is of great help to keep this channel going. And you too can be part of the story, go check out our page on patreon, the link is in the info below.

Let us then have a look at what scientists have found.

Researchers from Carnegie Mellon and other American universities have done a very interesting series of experiments using fMRI. In the first one, they put eleven trial participants in the MRI machine and showed them a word on a screen. The participants were asked to think of the concept related to a noun, for example an apple, a cat, a refrigerator, and so on. Then they gave the brain scans of 10 of these people to an artificially intelligent software, together with the word that the people were prompted with. The AI looked for patterns in the brain activity that correlated with the words, and then guessed what the 11th person was thinking of from the brain scan alone. The program guessed correctly about three quarters of the time.

That’s not particularly great, but it *is better than chance – it’s a proof of principle. And along the way the researchers made a very interesting finding. The study had participants whose first language was either English or Portuguese but their brain signature was independent of that. Indeed, the researchers found that in the brain, the concept encoded by a word doesn’t have much to do with the word itself. Instead, the brain encodes the concept by assigning different attributes to it. They have identified three of these attributes:

1) Eating related. This brain pattern activates for words like “apple”, “tomato” or “lettuce”
2) Shelter related. This pattern activates for example for “house”, “closet”, or “umbrella”, and
3) A body-object interaction. For example, if the concept is “pliers” the brain also activates the part representing your hand using the pliers.

This actually allows the computer to predict to some extent how the signal of a concept will look like even if the computer hasn’t seen data on that before. The researchers checked this by combining different concepts to sentences such as “The old man threw the stone into the lake”. Out of 240 possible sentences, the computer could pick the right one in eighty-three percent of cases. It is not that the computer can tell the whole sentence but it knows its basic components, it knows the semantic elements.

The basic finding of this experiment, that the brain identifies concepts by a combination of attributes, has been confirmed by other experiments. For example, another 2019 study, which also used fMRIs asked participants to think of different animals and found that the brain roughly classifies them by attributes like size, intelligence, and habitat.

In the last decade there have also been several attempts to find out what a person sees from their brain activity. For example, in 2017 a team from Kyoto University published a paper in which they used deep learning – so, artificial intelligence again – to find out what someone was seeing from their fMRI signal. They trained the software to recognize general aspects of the image, like shapes, contrast, faces, etc. You can judge the results for yourself. Here you see the actual images that the trial participants looked at, and here the reconstruction by the artificial intelligence – I find it really impressive.

What about speech or text? In April 2019, researchers from UCSF published a paper in Nature reporting they had successfully converted brain activity directly to speech. They worked with epilepsy patients that already had electrodes on their brain surface for treatment. What the researchers looked for were the motor signals that correspond to the sounds in speech, like the tongue, jaw, lips, and so on. Again, they let a computer figure out how to map the brain signal to speech. What you are about to hear is one of the participants reading a sentence and then what the software recreated just from the brain activity.

That’s pretty good, isn’t it? Unfortunately, it took weeks to decode the signals with that quality, so it’s rather useless in practice. But a new study that appeared just a few weeks ago has made a big leap forward for using brain-to text software by looking not at the movements related to producing sounds, but at the movements that come with handwriting.

The person who they worked with is paralyzed from the neck down and has electrodes implanted on his brain already. He was asked to imagine writing the letters of the alphabet, which was used to train the software, and later the AI could reproduce the text from brain activity when the subject imagined writing whole sentences. And, it could do that in real time. That allowed the paralyzed man to text at a speed of about 90 characters per minute, which is quite similar to what able-bodied people reach with text-messaging, about 135 characters. The AI was able to identify characters with over 94% accuracy, and with autocorrect that went up to 99%. So, as you can see, on the side of signal analysis, research has progressed quite rapidly in the past couple of years. But for technological applications the problem is that fMRIs are impractical, EEGs aren’t precise enough, and not everyone wants to have a USB port fused to their brain. Are there any other options?

Well, one thing that researchers have done is to genetically modify zebrafish larvae so that their neurons are fluorescent when active. That way you can measure brain activity non-invasively. And that’s nice, but even if you did that with humans, there’s still the skull in the way, so that doesn’t seem very promising.

More promising is an approach pursued by NASA which is to develop an infrared system to monitor brain activity. That still requires users to wear sensors around their head but it’s non-invasive. And several teams of scientists are trying to monitor brain activity by combining different non-invasive measurements: electrical and ultrasound and optical. The US military, for example, has put 104 million dollars into the Next-generation Nonsurgical Neurotechnology Program, or N cube for short which has the aim of controlling military drones.

We live in a momentous period in the history of human development. It’s the period when humans leave behind the idea that conscious thought is outside of science. So, all of a sudden, we can develop technologies to aid the conversion of thought into action. I find this incredibly interesting. I expect much to happen in this field in the coming years, and will update you from time to time, so don’t forget to subscribe.