Participants controlled the robot with their brain. Specifically, they had to focus their attention on one of three arrows - forward, left, and right - shown on the screen.
None of the elements of this process are new - real time fMRI has been around for a few years, so has machine learning to decode brain activation - but it's the first time they've been put together in this way.
And it's pretty awesome. The participants were able to guide their 'avatar' around a room to visit a number of target locations. They weren't perfectly accurate, and it took 10 or 15 minutes to navigate a few meters of ground... but it worked.
However... were they really using their minds, or just their eyes?
This is my main concern about this paper: participants were told to keep their eyes focussed on the middle of the screen and just mentally focus on the arrows to give commands. If they did indeed keep their eyes entirely stationary, then the patterns of brain activation would indeed represent pure 'thoughts'.
But if they were moving their eyes slightly (even unconsciously), the interpretation would be rather different. Moving their eyes would change the pattern of light hitting their retina, and this would be expected to change brain activation in the visual system of the brain.
So, maybe the fancy fMRI decoding system wasn't reading their mind, it was just acting as an elaborate means of tracking eye movements - which would be much less interesting. If you want to control a robot with your eyes, there are cheaper ways.
Andersson et al acknowledge this issue, and they claim, for various reasons, that this probably wasn't what happened here - but they didn't measure eye movements directly, so it does remain a worry. Eye tracking devices suitable for fMRI are widely available but this study used an ultra-powerful 7 Tesla scanner which, the authors say, made it impossible. So there's more work to be done here.
Andersson P, Pluim JP, Viergever MA, and Ramsey NF (2012). Navigation of a Telepresence Robot via Covert Visuospatial Attention and Real-Time fMRI. Brain topography PMID: 22965825
12 comments:
Attention strongly modulates fMRI signals in early visual areas, so the principle is sound. That is, if you simply attend to your left visual field the fMRI signals in that patch will increase, even if your eyes don't move. (There was a recent review on attention & visual areas, will see if I can find it to post later. It contains all the primary refs.)
I'd wager the bulk of the success of the machine learning algo is driven by early visual areas, too, in part because it's a large patch of cortex and the signals are large relative to those available from other brain regions. Someone called "anonymous" will hopefully pitch in with more details about these issues.
If they really fixated on the centre of the screen (which presumably changed constantly because of the robot's movement?), eye movements are unlikely to account for this in my opinion. They would have had to make fairly considerable eye movements for this to affect the signal in such a consistent way. But without eye tracking it's of course possible that this is what they did?
What they could have done instead of course is to use different signals for the three movements, such as mental imagery vs arithmetic vs another task. Many real-time fMRI experiments use such tasks so I don't see why this wouldn't have been possible here. But naturally, attending on the arrows is probably the most intuitive way to control an avatar.
Even more intuitive would of course be if they could read out brain activations associated with imagining to walk left, right and forward. That would be by far the best computer-brain interface (and then they should make it portable as it's probably cumbersome to only be able to do this from inside a 7T scanner...)
The brain is a machine for moving a body in the world. It doesn't matter if it's eye movements or 'pure thought'. What if the 'pure thought' is an eye movement signal plus strong inhibitory modulation preventing the eye movement from actually being made? What makes this 'pure thought' with the modulation but 'much less interesting' without the modulation? It's a false dichotomy. Accurate decoding of brain activity is cool whether or not there is attendant movement. What would be less interesting is if there were movement artifacts meaning brain activity was actually not driving the signal. That doesn't seem likely here though.
Having my warped brain controlling a robot isn't a wise thing. I guess that goes for most if one follows the news recently
We made a similar BCI for communication two years ago. We verified that the classifier used information from the motor cortex.
http://www.youtube.com/watch?v=QIJDGlM3uiE&feature=plcp
This work was presented at the International Conference on Pattern Recognition (ICPR) in 2010
http://liu.diva-portal.org/smash/get/diva2:297936/FULLTEXT01
Hi all,
As the first author of the discussed paper I would first very much like to thank you for your interest. Always nice to see that someone actually reads your work :)
I figured I could comment on some of the issues, and if you have any questions, please feel free to ask.
@Neuroskeptic:
As a minor detail (and of no importance...) I could mention that though I have done my work in the Netherlands, I am actually Swedish.
You write that "they could control the robot just by thinking about the correct arrow". This is not really true, it is the visuospatial attention that needs to be shifted towards the arrow. However, since you do indeed write this in one of the earlier sentences I am sure you are aware of this.
The main novelty of our study is the BCI control paradigm based on visuospatial attention, independent of both presented stimuli and eye-movements.
Now, the question was raised about the absence of eye-movements. This is indeed an important issue. Since there are no eye-trackers approved for 7 Tesla we could not record live eye-movements.
Despite the fact that it has been shown in many, many studies that people have no problem moving the spatial attention without moving the eyes, it would of course be nice to have the data.
However, it is quite clear from the activation patterns that the effect we measure does not come from eye-movements. If you move your eyes towards one of the targets you induce intensity changes at the location of all three targets, plus in the complete foveal area (where the video is shown). This gives a very "messy" activation pattern that looks quite different from when only attending the target. Moreover, since the video shows different images at all times there would not be a consistent intensity change in that region. You can see a comparison between the two types of activation patterns in an earlier paper [1] (open access). There we only used two attention targets, and thus the effect would be even larger in the current study. In that paper we also show examples of offline EOG measurements recorded during the task.
I can tell you from my own experience that when you are in the scanner to control the robot, you really do not want to move the eyes since you immediately lose the control. (Please note that the previous sentence is obviously not meant as any kind of proof!)
If you are interested you can find a similar study of ours in Journal of Neural Engineering [2].
@practiCal fMRI:
Though the machine learning does improve the classification, it is almost as good using univariate classification. In [2] we classify attention to four directions online with an accuracy of 80%. I would be happy to send you the paper if you are interested.
I can also mention that we do not see much effect in V1. It is mainly V2, V3, V3A/B and more parietal areas that gets activated. This is also what has been found in earlier studies.
@Nitpicker:
Indeed there would have to be large eye movements. Such movements would not be done unconsciously, and I doubt that all subjects tried to deceive me :).
The reason we did not use other types of control tasks is that the study was done with the purpose of demonstrating 'COVISA' based control. There are many problems with motor based signals, but I will not go into that now. Working memory based control is actually something that we are working on.
[1] "Real-Time Decoding of Brain Responses to Visuospatial Attention Using 7T fMRI", PLoS ONE, 6(11), e27638, (2011)
[2] "Real-time decoding of the direction of covert visuospatial attention",
Journal of Neural Engineering, 9(4), (2012)
Thanks again for your comments. I hope you don't mind my long reply...
Patrik Andersson
This is a very interesting article, I never knew we had the technology to control a robot with merely our thoughts. Although it is stated in the article that people may have been moving their eyes (even unconsciously) it would be interesting to find out which part of the brain would be used the most in the process of controlling this robot with pure thought. It probably just depends on the individual person's brain, but there could be many parts of the brain involved. The best thing to compare this to would be driving a car, and what parts of the brain are used the most during this process. The primary motor cortex would deal with the physical part of driving, but controlling the robot only uses thoughts, so much use of this part of the brain is unlikely, but always a possibility. The primary visual cortex, which deals with visual perception, should also take part in controlling the robot. Because "driving" this robot with pure thought involves the mental process of deciding which way to go (and purely thinking in that direction will turn the robot whatever direction you think of), I feel that the most significant part of the brain used in controlling the robot would be the frontal lobes, which deals with perception, planning, and decision making.
Patrik: Thanks very much for the comment!
I think that the patients in this experiment were using their minds more than their eyes because they had to keep their eyes in the same place. Since they kept their eyes in the same place they must have been moving it with their mind. They couldn’t move their eyes because I believe that, that would create more activity in one’s brain just by the dilating or constricting of the pupil adjusting to the light or to the things one sees through his or her peripheral vision. The Machine Learning Algorithm is not technical enough to pick up a variety of brain activates, so if one keeps his or her eyes in the same position resulting in the least amount of brain activity, then the machine learning algorithm will be able to recognize the pattern and tell the robot what to do.
Wait... control robots with your brain? Seems the future is almost here. Well, the brain is a mysterious and awe inspiring organic "computer". Like space, we have yet to fully delve into what it contains and can truly do. So, i look forward welcoming our "metallic" future.
I think this could be done if we were able to tap into the brain(cerebellum) or the central nervous system or the peripheral nervous system. All of these which controls the bodies movement.
thanks for sharing.
Post a Comment