Can We Rely on fMRI?

Craig Bennett (of Prefrontal.org) and Michael Miller, of dead fish brain scan fame, have a new paper out: How reliable are the results from functional magnetic resonance imaging?


Tal over at the [citation needed] blog has an excellent in-depth discussion of the paper, and Mind Hacks has a good summary, but here's my take on what it all means in practical terms.

Suppose you scan someone's brain while they're looking at a picture of a cat. You find that certain parts of their brain are activated to a certain degree by looking at the cat, compared to when they're just lying there with no picture. You happily publish your results as showing The Neural Correlates of Cat Perception.

If you then scanned that person again while they were looking at the same cat, you'd presumably hope that exact same parts of the brain would light up to the same degree as they did the first time. After all, you claim to have found The Neural Correlates of Cat Perception, not just any old random junk.

If you did find a perfect overlap in the area and the degree of activation that would be an example of 100% test-retest reliability. In their paper, Bennett and Miller review the evidence on the test-retest reliability of fMRI studies. They found 63 of them. On average, they found that the reliability of fMRI falls quite far short of perfection: the areas activated (clusters) had a mean Dice overlap of 0.476, while the strength of activation was correlated with a mean ICC of 0.50.

But those numbers, taken out of context, do not mean very much. Indeed, what is a Dice overlap? You'll have to read the whole paper to find out, but even when you do, they still don't mean that much. I suspect this is why Bennett and Miller don't mention them in the Abstract of the paper, and in fact they don't spend more than a few lines discussing them at all.

A Dice overlap of 0.476 and an ICC of 0.50 are what you get if average over all of the studies that anyone's done looking at the test-retest reliability of any particular fMRI experiment. But different fMRI experiments have different reliabilities. Saying that the average reliability of fMRI is 0.5 is rather like saying that the mean velocity of a human being is 0.3 km per hour. That's probably about right, averaging over everyone in the world, including those who are asleep in bed and those who are flying on airplanes - but it's not very useful. Some people are moving faster than others, and some scans are more reliable than others.


Most of this paper is not concerned with "how reliable fMRI is", but rather, with how to make any given scanning experiment more reliable. And this is an important thing to write about, because even the most optimistic cognitive neuroscientist would agree that many fMRI results are not especially reliable, and as Bennett and Miller say, reliability matters for lots of reasons:

Scientific truth. While it is a simple statement that can be taken straight out of an undergraduate research methods course, an important point must be made about reliability in research studies: it is the foundation on which scientific knowledge is based. Without reliable, reproducible results no study can effectively contribute to scientific knowledge.... if a researcher obtains a different set of results today than they did yesterday, what has really been discovered?
Clinical and Diagnostic Applications. The longitudinal assessment of changes in regional brain activity is becoming increasingly important for the diagnosis and treatment of clinical disorders...
Evidentiary Applications. The results from functional imaging are increasingly being submitted as evidence into the United States legal system...
Scientific Collaboration. A final pragmatic dimension of fMRI reliability is the ability to share data between researchers...

So what determines the reliability of any given fMRI study? Lots of things. Some of them are inherent to the nature of the brain, and are not really things we can change: activation in response to basic perceptual and motor tasks is probably always going to be more reliable than activation related to "higher" functions like emotions.

But there are lots of things we can change. Although it's rarely obvious from the final results, researchers make dozens of choices when designing and analyzing an fMRI experiment, many of which can at least potentially have a big impact on the reliability of their findings. Bennett and Miller cover lots of them:

voxel size... repetition time (TR), echo time (TE), bandwidth, slice gap, and k-space trajectory... spatial realignment of the EPI data can have a dramatic effect on lowering movement-related variance ... Recent algorithms can also help remove remaining signal variability due to magnetic susceptibility induced by movement... simply increasing the number of fMRI runs improved the reliability of their results from ICC = 0.26 to ICC = 0.58. That is quite a large jump for an additional ten or fifteen minutes of scanning...

The details get extremely technical, but then, when you do an fMRI scan you're using a superconducting magnet to image human neural activity by measuring the quantum spin properties of protons. It doesn't get much more technical.

Perhaps the central problem with modern neuroimaging research is that it's all too easy for researchers to write off the important experimental design issues as "merely" technicalities, and just put some people in a scanner using the default scan sequence and see what happens. This is something few fMRI users are entirely innocent of, and I'm certainly not, but it is a serious problem. As Bennett and Miller point out, the devil is in the technical details.

The generation of highly reliable results requires that sources of error be minimized across a wide array of factors. An issue within any single factor can significantly reduce reliability. Problems with the scanner, a poorly designed task, or an improper analysis method could each be extremely detrimental. Conversely, elimination of all such issues is necessary for high reliability. A well maintained scanner, well designed tasks, and effective analysis techniques are all prerequisites for reliable results.

Bennett CM, Miller MB. (2010). How reliable are the results from functional magnetic resonance imaging? Annals of the New York Academy of Sciences

Life Without Serotonin

Via Dormivigilia, I came across a fascinating paper about a man who suffered from a severe lack of monoamine neurotransmitters (dopamine, serotonin etc.) as a result of a genetic mutation: Sleep and Rhythm Consequences of a Genetically Induced Loss of Serotonin


Neuroskeptic readers will be familiar with monoamines. They're psychiatrists' favourite neurotransmitters, and are hence very popular amongst psych drug manufacturers. In particular, it's widely believed that serotonin is the brain's "happy chemical" and that clinical depression is caused by low serotonin while antidepressants work by boosting it.

Critics charge that there is no evidence for any of this. My own opinion is that it's complicated, but that while there's certainly no simple relation between serotonin, antidepressants and mood, they are linked in some way. It's all rather mysterious, but then, the functions of serotonin in general are; despite 50 years of research, it's probably the least understood neurotransmitter.

The new paper adds to the mystery, but also provides some important new data. Leu-Semenescu et al report on the case of a 28 year old man, with consanguineous parents, who suffers from a rare genetic disorder, sepiapterin reductase deficiency (SRD). SRD patients lack an enzyme which is involved, indirectly, in the production of the monoamines serotonin and dopamine, and also melatonin and noradrenaline which are produced from these two. SRD causes a severe (but not total) deficiency of these neurotransmitters.

The most obvious symptoms of SRD are related to the lack of dopamine, and include poor coordination and weakness, very similar to Parkinson's Disease. An interesting feature of SRD is that these symptoms are mild in the morning, worsen during the day, and improve with sleep. Such diurnal variation is also a hallmark of severe depression, although in depression it's usually the other way around (better in the evening).

The patient reported on in this paper suffered Parkinsonian symptoms from birth, until he was diagnosed with dystonia at age 5 and started on L-dopa to boost his dopamine levels. This immediately and dramatically reversed the problems.

But his serotonin synthesis was still impaired, although doctors didn't realize this until age 27. As a result, Leu-Semenescu et al say, he suffered from a range of other, non-dopamine-related symptoms. These included increased appetite - he ate constantly, and was moderately obese - mild cognitive impairment, and disrupted sleep:

The patient reported sleep problems since childhood. He would sleep 1 or 2 times every day since childhood and was awake during more than 2 hours most nights since adolescence. At the time of the first interview, the night sleep was irregular with a sleep onset at 22:00 and offset between 02:00 and 03:00. He often needed 1 or 2 spontaneous, long (2- to 5-h) naps during the daytime.

After doctors did a genetic test and diagnosed SRD, they treated him with 5HTP, a precursor to serotonin. The patient's sleep cycle immediately normalized, his appetite was reduced and his concentration and cognitive function improved (although that may have been because he was less tired). Here's his before and after hypnogram:

Disruptions in sleep cycle and appetite are likewise common in clinical depression. The direction of the changes in depression varies: loss of appetite is common in the most severe "melancholic" depression, while increased appetite is seen in many other people.

For sleep, both daytime sleepiness and night-time insomnia, especially waking up too early, can occur in depression. The most interesting parallel here is that people with depression often show a faster onset of REM (dreaming) sleep, which was also seen in this patient before 5HTP treatment. However, it's not clear what was due to serotonin and what was due to melatonin because melatonin is known to regulate sleep.

Overall, though, the biggest finding here was a non-finding: this patient wasn't depressed, despite having much reduced serotonin levels. This is further evidence that serotonin isn't the "happy chemical" in any simple sense.

On the other hand, the similarities between his symptoms and some of the symptoms of depression suggest that serotonin is doing something in that disorder. This fits with existing evidence from tryptophan depletion studies showing that low serotonin doesn't cause depression in most people, but does re-activate symptoms in people with a history of the disease. As I said, it's complicated...

Smaranda Leu-Semenescu et al. (2010). Sleep and Rhythm Consequences of a Genetically Induced Loss of Serotonin Sleep, 33 (03), 307-314

Vote Neuroskeptic 2010!

So, wow. Neuroskeptic is a finalist for the ResearchBlogging.org Awards, in the categories of Research Blog of the Year and Post of the Year.

I'm honoured to have made it into the Top 10 alongside great blogs such as Neurophilosophy, Not Exactly Rocket Science and Science-Based Medicine, all of which I read before I began blogging and which helped inspired me to start. Thanks to everyone who nominated me, and of course, to the judges,

Anyway, the winners will be chosen by a vote of ResearchBlogging.org members. Voting will be open for one week from March 4 - i.e. later today - to March 11. So if you think science blogging is important, and you think that Neuroskeptic deserves the $1000 top prize, please vote. Or if you wouldn't trust me with that kind of money, you could at least vote for the Best Post prize of $50 for my post on the voodoo correlations in fMRI issue (post). Or you could even do both.

Don't forget to vote in the other categories too. The finalists for Best Blog -- Health, Best Blog -- Psychology, and Best Blog -- Neuroscience are all excellent, so it'll be a tough choice.

Is Your Brain A Communist?

Capitalists beware. No less a journal than Nature has just published a paper proving conclusively that the human brain is a Communist, and that it's plotting the overthrow of the bourgeois order and its replacement by the revolutionary Dictatorship of the Proletariat even as we speak.

Kind of. The article, Neural evidence for inequality-averse social preferences, doesn't mention the C word, but it does claim to have found evidence that people's brains display more egalitarianism than people themselves admit to.

Tricomi et al took 20 pairs of men. At the start of the study, both men got a $30 payment, but one member of each pair was then randomly chosen to get a $50 bonus. Thus, one guy was "rich", while the other was "poor". Both men then had fMRI scans, during which they were offered various sums of money and saw their partner being offered money too. They rated how "appealing" these money transfers were on a 10 point scale.

What happened? Unsurprisingly both "rich" and "poor" said that they were pleased at the prospect of getting more cash for themselves, the poor somewhat more so, but people also had opinions about payments to the other guy:

the low-pay group disliked falling farther behind the high-pay group (‘disadvantageous inequality aversion’), because they rated positive transfers to the high-pay participants negatively, even though these transfers had no effect on their own earnings. Conversely, the high-pay group seemed to value transfers [to the poor person] that closed the gap between their earnings and those of the low-pay group (‘advantageous inequality aversion’)

What about the brain? When people received money for themselves, activity in the ventromedial prefrontal cortex (vmPFC) and the ventral striatum correlated with the size of their gain.

However, when presented with a payment to the other person, these areas seemed to be rather egalitarian. Activity rose in rich people when their poor colleagues got money. In fact, it was greater in that case than when they got money themselves, which means the "rich" people's neural activity was more egalitarian than their subjective ratings were. Whereas in "poor" people, the vmPFC and the ventral striatum only responded to getting money, not to seeing the rich getting even richer.


The authors conclude that this

indicates that basic reward structures in the brain may reflect even stronger equity considerations than is necessarily expressed or acted on at the behavioural level... Our results provide direct neurobiological evidence in support of the existence of inequality-averse social preferences in the human brain.

Notice that this is essentially a claim about psychology, not neuroscience, even though the authors used neuroimaging in this study. They started out by assuming some neuroscience - in this case, that activity in the vmPFC and the ventral striatum indicates reward i.e. pleasure or liking - and then used this to investigate psychology, in this case, the idea that people value equality per se, as opposed to the alternative idea, that "dislike for unequal outcomes could also be explained by concerns for social image or reciprocity, which do not require a direct aversion towards inequality."

This is known as reverse inference, i.e. inference from data about the brain to theories about the mind. It's very common in neuroimaging papers - we've all done it - but it is problematic. In this case, the problem is that the argument relies on the idea that activity in the vmPFC and ventral striatum is evidence for liking.

But while there's certainly plenty of evidence that these areas are activated by reward, and the authors confirmed that activity here correlated with monetary gain, that doesn't mean that they only respond to reward. They could also respond to other things. For example, there's evidence that the vmPFC is also activated by looking at angry and sad faces.

Or to put it another way: seeing someone you find attractive makes your pupils dilate. If you were to be confronted by a lion, your pupils would dilate. Fortunately, that doesn't mean you find lions attractive - because fear also causes pupil dilation.

So while Tricomi et al argue that people, or brains, like equality, on the basis of these results, I remain to be fully convinced. As Russell Poldrack noted in 2006

caution should be exercised in the use of reverse inference... In my opinion, reverse inference should be viewed as another tool (albeit an imperfect one) with which to advance our understanding of the mind and brain. In particular, reverse inferences can suggest novel hypotheses that can then be tested in subsequent experiments.

Tricomi E, Rangel A, Camerer CF, & O'Doherty JP (2010). Neural evidence for inequality-averse social preferences. Nature, 463 (7284), 1089-91 PMID: 20182511

The Crazies

I just watched The Crazies, a remake of Romero's 1973 original of the same name, about a small town struck by an outbreak of insanity following a biological weapon accident. It's not for the faint of heart: I was unsettled by a number of the scenes and I watch a lot of horror movies.

Which is to say, it's excellent. It maintains a high pitch of tension through the whole 100 minutes, something that a lot of horror doesn't manage. All too often, I find, a movie will start out scary enough, but then by some point about half way through it's effectively turned into an action movie.

This happens when the nature of the monster/killer/zombies have been revealed and all the protagonists have to do is fight it out - with the uncertainty gone, the horror goes, too. Without giving too much away, The Crazies avoids this trap. (The last great horror movie I saw, Paranormal Activity, does too, although in a very different way).

Of course the real reason I liked this movie is that it's got some neuroscience. The Crazies is (spoilers) about an engineered virus that infects the brain. Early symptoms include fever, blank stares, flattened emotions and stereotypies. This then progresses, over the course of about 48 hours, to psychopathic aggression, at least in some cases, although other victims just become confused. The "crazies" are somewhat like zombies - they have a Zombie Spectrum Disorder, one might say - but they retain enough of their personality and intelligence to be capable of much more elaborate and calculating violence than the average braaaaaaains-muncher, which is what makes them so disturbing.

Could a virus do that? Rabies, notoriously, causes aggression in animals and humans, although the incubation period is weeks rather than days, and aggression is only one of many neurological symptoms of the disease. But maybe an engineered virus could achieve a more specific effect if it was able to selectively infect the area of the brain reported on in this rather scary paper:

The authors report a patient with advanced PD, successfully treated by bilateral stimulation of the subthalamic nucleus, who developed acute transient aggressive behavior during intraoperative electrical test stimulation. The electrode responsible for this abnormal behavior was located within the lateral part of the posteromedial hypothalamic region (triangle of Sano). The authors suggest that affect can be dramatically modulated by the selective manipulation of deep brain structures.