Another Scuffle In The Coma Ward

It's not been a good few weeks for Adrian Owen and his team of Canadian neurologists.

Over the past few years, Owen's made numerous waves, thanks to his claim that some patients thought to be in a vegetative state may, in fact, be at least somewhat conscious, and able to respond to commands. Remarkable if true, but not everyone's convinced.

A few weeks ago, Owen et al were criticized over their appearance in a British TV program about their use of fMRI to measure brain activity in coma patients. Now, they're under fire from a second group of critics over a different project.

The new bone of contention is a paper published in 2011 called Bedside detection of awareness in the vegetative state. In this report, Owen and colleagues presented EEG results that, they said, show that some vegetative patients are able to understand speech.

In this study, healthy controls and patients were asked to imagine performing two different actions: moving their hand, or their toe. Owen et al found that it was possible to distinguish between the 'hand' and 'toe'-related patterns of brain electrical activity. This was true of most healthy control subjects, as expected, but also of some - not all - patients in a 'vegetative' state.

The skeptics aren't convinced, however. They reanalyzed the raw EEG data and claim that it just doesn't prove anything.

This image shows that in a healthy control, EEG activity was "clean" and generally normal. However in the coma patient, the data's a mess. It's dominated by large slow delta waves - in healthy people, you only see those during deep sleep - and there's also a lot of muscle artefacts which can be seen as 'thickening' of the lines.

These don't come from the brain at all, they're just muscle twitches. Crucially, the location and power of these twitches varied over time (as muscle spikes often do).

This wouldn't necessarily be a problem, the critics say, except that the statistics used by Owen et al didn't control for slow variations over time i.e. of correlations between consecutive trials (non-independence). If you do take account of these, there's no statistically significant evidence that you can distinguish the EEG associated with 'hand' vs 'toe' in any patients.

However, in their reply, Owen's team say that:

their reanalysis only pushes two of our three positive patients to just beyond the widely accepted p=0.05 threshold for significance - to p=0.06 and p=0·09, respectively. To dismiss the third patient, whose data remain significant, they state that the statistical threshold for accepting command-following should be adjusted for multiple comparisons... but we know of no groups in this field who routinely use such a conservative correction with patient data, including the critics themselves.

I have to say that, statistical arguments aside, the EEGs from the patients just don't look very reliable, largely because of those pesky muscle spikes. A new method for removing these annoyances has just been proposed... I wonder if that could help settle this?

Goldfine, A., Bardin, J., Noirhomme, Q., Fins, J., Schiff, N., and Victor, J. (2013). Reanalysis of "Bedside detection of awareness in the vegetative state: a cohort study" The Lancet, 381 (9863), 289-291 DOI: 10.1016/S0140-6736(13)60125-7

A Scuffle In The Coma Ward

A couple of months ago, the BBC TV show Panorama covered the work of a team of neurologists (led by Prof. Adrian Owen) who are pioneering the use of fMRI scanning to measure brain activity in coma patients.

The startling claim is that some people who have been considered entirely unconscious for years, are actually able to understand speech and respond to requests - not by body movements, but purely on the level of brain activation.

However, not everyone was impressed. A group of doctors swiftly wrote a critical response, published in the British Medical Journal as fMRI for vegetative and minimally conscious states: A more balanced perspective

The Panorama programme... failed to distinguish clearly between vegetative vs. minimally conscious states, and gave the impression that 20% of patients in a vegetative state show cognitive responses on fMRI.

There are important differences between the two states. Patients in a vegetative state have no discernible awareness of self and no cognitive interaction with their environment. Patients in a minimally conscious state show evidence of interaction through behaviours...

The programme presented two patients said to be in a “vegetative state” who showed evidence of cognitive interaction on assessment using fMRI but the clinical methods used for the original diagnosis were not stated. In both cases, family members clearly reported that the patient made positive but inconsistent behavioural responses to questions... one of these patients was filmed responding to a question from his mother by raising his thumb and the other seemed to turn his head purposefully.

So Panorama stands accused of passing off patients who were really minimally conscious, as being in a vegetative state. To see signs of understanding on brain scans from the latter would be truly amazing because it would be the first evidence that they weren't, well, vegetative.

However if they were 'merely' minimally conscious patients, it's not as interesting, because we already knew they were capable of making responses.

Now the Panorama team - and Professor Owen - have replied in a BMJ piece of their own. Given that they're charged with  misleading journalism and sloppy medicine, they're understandably a bit snarky:

Just by viewing this one hour documentary the authors felt able to discern that both the patients “said to be in a vegetative state” are “probably” minimally conscious... One of these patients, Scott, has had the same neurologist for more than a decade. Professor Young, who appeared in the film, made it clear that Scott had appeared vegetative in every assessment...

The fact that these authors took Scott’s fleeting movement, shown in the programme, to indicate a purposeful (“minimally conscious”) response shows why it is so important that the diagnosis is made in person, by an experienced neurologist, using internationally agreed criteria.

In other words, they were vegetative, and the critics who said otherwise, on the basis of some TV footage, were being silly.

In other words...it's on.

Turner-Stokes L, Kitzinger J, Gill-Thwaites H, Playford ED, Wade D, Allanson J, Pickard J, & Royal College of Physicians' Prolonged Disorders of Consciousness Guidelines Development Group (2012). fMRI for vegetative and minimally conscious states. BMJ (Clinical research ed.), 345 PMID: 23190911

Walsh F, Simmonds F, Young GB, & Owen AM (2013). Panorama responds to editorial on fMRI for vegetative and minimally conscious states. BMJ (Clinical research ed.), 346 PMID: 23298817

Notorious Paedophile Reads Neuroskeptic

There's been controversy in the UK over an article published in the Guardian that's regarded as being pro-child abuse.

In particular, the newspaper has taken flak for quoting Tom O'Carroll, a self-confessed paedophile and advocate for the right of people to be one. Amongst other things he's written a book about Michael Jackson. At least this week, until the next one comes along, he is Britain's most notorious paedophile.

Now oddly enough, I recently had an encounter with O'Carroll, although I didn't know who he was at the time. Here's the tale...

A few weeks ago, I got an out-of-the-blue email from a Neuroskeptic reader (as happens often), from someone saying he'd tried to leave a comment on this post but it was rejected for being too long.

O'Carroll (for it was he) wanted to know whether I agreed with his reservations about some research on "Cerebral white matter deficiencies in pedophilic men", and specifically on this statement by the paper's author:

One must consider carefully whether the brain differences we detected [i.e. reduced white matter volume in particular areas] cause pedophilia or whether some aspect of being pedophilic caused the brain differences...

Although it is now known that certain brain structures respond to environmental stimulation, such as the motor cortex, there is no evidence that such stimulation causes any changes in the superior fronto-occipital fasciculus or right arcuate fasciculus (the brain regions in which pedophiles and nonpedophiles differ).

Moreover, the brain regions we identified are extremely large, and no previous research has ever found changes in such large regions of the brain. As an analogy, physical exercise will generally stimulate one’s muscle tissue to grow, but one would not grow an extra arm; neurological changes occur only in a very specific manner.

O'Carroll thought that this is a misleading analogy, because the brain could be plastic in ways we don't yet understand. I agreed. We just don't know enough about the brain, yet, to say that the observed white matter changes can't possibly be responses to experiences.

Several recent studies found that experience and learning can change adult human white matter structure. The changes observed were fairly small, but then these studies were fairly short, so they don't define the upper limit of what's possible over time.

These results are not without critics of their own, and I'm on the fence about whether white matter is plastic at all, but my point is, it's an open question. So comparing the idea of white matter plasticity to 'growing an extra arm' is overstatement - and it would be, whether the topic was paedophilia or anything else.

Now, as I said, I hadn't heard of Tom O'Carroll at the time, and I assumed he had a purely academic interest in the matter, as an piece of oversold neuroscience. But now, thanks to the Guardian drama, I realize that...

Britain's most notorious paedophile reads Neuroskeptic.

Hmm.

On that macabre note, I've often wondered whether James Eagan Holmes, the Aurora, Colorado Batman shooter, ever visited this blog. It's possible: he was a neuroscience undergraduate and later PhD student over the time Neuroskeptic's been going.

There have been 574 visits from Aurora, Colorado, where Holmes was doing his PhD, since 2008. I'll never know whether he was one of them, but the idea that he might have been is pretty creepy.

Flawed Statistics Make Almost Everyone's Brain "Abnormal"

A popular method for detecting abnormalities in the shape and size of individual brains is seriously flawed, and is almost guaranteed to find 'differences' even in normal people.

So say Italian neuroscientists Scarpazza and colleagues in an important new report: Very high false positive rates in single case Voxel Based Morphometry.

Voxel Based Morphometry (VBM) is a way of analyzing brain scans to detect structural differences. It's most commonly used to compare groups of brains to find average differences, but some neuroscientists have started using VBM to check for abnormalities in a single brain. Scarpazza et al list 34 pieces of research about that, including 13 since 2010.

So it would suck if there were a problem with individual VBM... but there is. This pic tells the tale:

The authors took 200 normal brains and compared each one of them in turn to a control group of 16 normal brains. Because all of them were healthy, the comparisons ought to show no significant differences.

The technique was set up so that, in theory, only 5% of the brains should have been wrongly labelled as containing an abnormality. But in fact, a full 93.5% of the normal brains gave at least one false positive.

So 5% is more like the rate of not being wrong. Oops.

The image shows that in some brain areas, almost 25% of the normal brains were branded as 'abnormal' just in that region alone - the hotter the colour, the higher the proportion of false 'hits'. The top row is for false reports of brain volume increases, while the bottom row is decreases; false 'increases' were more common.

So what's going wrong? It's not entirely clear and several factors are probably at play, but the authors say that the main issue is that VBM makes the assumption of statistical normality which doesn't in fact hold.

Either way, it's a serious problem, and Scarpazza et al point to one especially worrying implication: some people have proposed using single-subject VBM in a legal context, to reinforce insanity pleas by showing subtle 'brain abnormalities' not obvious to the naked eye. Yet if this paper's right, such evidence could be entirely meaningless, almost guaranteed to give a positive result.

P.S. Last time I posted about this kind of analysis flaw, the internet went crazy because they didn't understand it. So just to be clear, this is not a problem for clinical scans - the kind you'd get to check whether you have a brain tumour.

Scarpazza, C., Sartori, G., De Simone, M., and Mechelli, A. (2013). When the single matters more than the group: Very high false positive rates in single case Voxel Based Morphometry NeuroImage DOI: 10.1016/j.neuroimage.2012.12.045

Am I Attacking Neuroscience?

A New York Times article just out says:

Neuroscience: Under Attack

Under attack by who?

Er... me. And the rest of the usual suspects:

A gaggle of energetic and amusing, mostly anonymous, neuroscience bloggers - including Neurocritic, Neuroskeptic, Neurobonkers and Mind Hacks - now regularly point out the lapses and folly contained in mainstream neuroscientific discourse. 

I had promised not to do any more self-referential posts, but this one wasn't my fault. Just when I thought I was out, they pull me back in.

Anyway, I'm pretty happy with how Neuroskeptic's presented in the article, but not entirely.

The headline is sensationalist - I don't see myself as attacking neuroscience and I don't think any of the others do either. We are trying to defend neuroscience against errors and misrepresentations. My ideal is The Sceptical Chymist, where skepticism helped, rather than undermined, chemistry.

But the job of a headline is to be sensationalist so that's OK. Most of the piece is very good. I'm all on board with this:

Meet the "neuro doubters". The neuro doubter may like neuroscience but does not like what he or she considers its bastardization by glib, sometimes ill-informed, popularizers.

Yet I can't quite go along with this:

A number of the neuro doubters are also humanities scholars who question the way that neuroscience has seeped into their disciplines, creating phenomena like neuro law, which, in part, uses the evidence of damaged brains as the basis for legal defense of people accused of heinous crimes, or neuroaesthetics, a trendy blend of art history and neuroscience.

Admittedly this wasn't directly aimed at me because I'm not a humanities scholar, but I believe that neuroaesthetics and neurolaw are absolutely valid - in theory.

I'm not defending any particular manifestation of those, and I've criticized quite a few. But in the abstract, I see nothing wrong with neuroscience helping to explain those things. It will be difficult in practice, but it's fine to try.

Autism Brain Scans Flawed? You Read It Here First

According to a piece in Nature today, a major line of research about autism might be seriously flawed:

One of the most popular and widely accepted theories on the cause of autism spectrum disorders attributes the condition to disrupted connectivity between different regions of the brain.

This 'connectivity hypothesis' claims that the social and cognitive abnormalities in people with autism can be explained by a dearth of connections between distant regions of the brain. Some flavours of this theory also predict more connections between nearby brain regions.

Recent studies, however, have found that when a person moves their head while undergoing functional magnetic resonance imaging (fMRI) - a method that maps how different neuroanatomical structures of the brain interact in real time, its functional connectivity - it looks like the neural activity observed in autism. That's a sobering discovery...

So the characteristic pattern of "abnormal connectivity" in autism might not be real: it might just reflect the fact that people with autism move around more during the scan.

A sobering idea, indeed... but not an entirely new one. I suggested it over a year ago:

Head motion affects estimates of functional connectivity. The more motion, the weaker the measured connectivity in long-range networks, while shorter range connections were stronger... Disconcertingly, this is exactly what's been proposed to happen in autism (although in fairness, not all the evidence for this comes from fMRI). This clearly doesn't prove that the autism studies are all dodgy, but it's an issue. People with autism, and people with almost any mental or physical disorder, on average tend to move more than healthy controls.

Ben Deen, and Kevin Pelphrey (2012). Perspective: Brain scans need a rethink Nature

Gene-Guided Antidepressants?

Over the past couple of years, "Big Pharma" has largely moved away from psychiatric drug development. This shift has been widely discussed.

But another trend has been happening over the same time period - or so it seems to me. This is the rise of small companies who offer techniques for diagnosing mental illness, or predicting which drugs will work best. Generally (it seems) partnerships between venture capitalists and psychiatry (ex-)researchers, these enterprises might be dubbed "Little Pharma".

The latest is a company called AssureRx Health, Inc. According to a paper just published, they offer

a pharmacogenomic algorithm designed to improve the safety and efficacy of prescribing antidepressant and antipsychotic medication... based on the genotyping of both copies of five genes.

From this, you end up with a report giving each drug a rating of green, yellow, or red (see above).

The price is not provided on their website.

According to the paper, they gave 26 depressed patients normal treatment at the discretion of their psychiatrist, while 25 got treatment guided by the AssureRx algorithm. It was non-randomized, and unblinded so there's a clear possibility of a placebo effect.

Anyway, the results were...

For the first 4 weeks of treatment, there was no difference between the two groups whatsoever in terms of depression symptom scores - they both improved. But then by week 8, the unguided patients abruptly got worse, while the AssureRx-guided ones continued to benefit. This is an unusual pattern of improvement in an antidepressant trial.

Previously, I wrote about another Little Pharma antidepressant prediction scheme. It used a  different approach, measuring brain electrical activity using a technique called "rEEG", rather than genetics. But the basic idea is the same... and so are the problems.

As I said last time:

There were two groups and they got entirely different sets of drugs. One group also got rEEG-based treatment personalization. That group did better, but that might have nothing to do with the rEEG: they might have done equally well if they'd just been assigned to [those drugs] by flipping a coin. We cannot tell, from these data, whether rEEG offered any benefits at all.

In the AssureRx paper, we can't even tell whether the two groups got different kinds of drugs, because the meds used aren't reported, but if they did differ then that would offer an alternative explanation for the differences in outcome: maybe the 'guidance' just recommended better drugs overall, with the genes being just a sideshow. Or maybe it's a placebo, as I said.

Moving on, I also wrote... 

What's curious is that it would have been very simple to avoid this issue. Just give everyone rEEG, but shuffle the assignments in the control group, so that everyone was guided by someone else's EEG. So you'd give control Patient 2 the drugs that Patient 1 should have got, and vice versa; swap 3 and 4, 5 and 6, etc.

This would be a genuinely controlled test of the personalization, because both groups would get the same kinds of drugs... and it would allow the trial to be double-blind: in this study the investigators knew which group people were in, because it was obvious from the drug choice...

It is odd that Little Pharma never seem to do such real vs. muddled prediction studies, as they'd be really informative as to whether their approach is a helpful innovation as opposed to an expensive, meaningless red herring. Hmm.

Hall-Flavin, D., Winner, J., Allen, J., Jordan, J., Nesheim, R., Snyder, K., Drews, M., Eisterhold, L., Biernacka, J., and Mrazek, D. (2012). Using a pharmacogenomic algorithm to guide the treatment of depression Translational Psychiatry, 2 (10) DOI: 10.1038/tp.2012.99

More on False Positive Neuroimaging

Back in June, I warned that the ever-increasing number of clever methods for analyzing brain imaging data could be a double-edged sword:

Recently, psychologists Joseph Simmons, Leif Nelson and Uri Simonsohn made waves when they published a provocative article called False-Positive Psychology - Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant.

It explained how there are so many possible ways to gather and analyze the results of a  simple psychology experiment that, even if there's nothing interesting really happening, it'll be possible to find some "significant" positive results purely by chance...

The problem's not just seen in psychology however, and I'm concerned that it's especially dangerous in modern neuroimaging research.

In a comment on that post, The Neurocritic pointed out that Michigan PhD student Joshua Carp had put forward the same argument in a conference presentation, several months previously.

Now Carp's published a paper on the topic: On the plurality of (methodological) worlds: estimating the analytic flexibility of fMRI experiments. It's free to access, so check it out.

Whereas I just talked the talk by listing lots of possible ways in which you could analyze a given set of data, Carp walked the walk, and actually did loads of analyses. He took a single dataset, the results of a simple experiment and looked at it in almost 7000 different ways. Each set of results was then thresholded to correct for multiple comparisons in 5 ways, for a grand total of 35,000 outputs.

The variants he considered ranged from how much smoothing to apply, to how to correct for head motion, and many more.

What happened? In a nutshell, the different options made a difference - and the variability was the largest in parts of the brain that were most activated (the "blobs" that lit up). In other words, analytic flexibility makes the most difference in the most interesting places. See the picture at the top.

The location of the maximum peak activation also varied. This is not unexpected, and not, in itself, that worrying - the great majority of the peaks clustered in a few small areas. However, it underlines that different options really can make a difference.

Carp concludes:

Nearly every voxel in the brain showed significant activation under at least one analysis pipeline. In other words, a sufficiently persistent researcher determined to find significant activation in virtually any brain region is quite likely to succeed...

If investigators apply several analysis pipelines to an experiment, and only report the analyses that support their hypotheses, then the prevalence of false positive results in the literature may far exceed the nominal rate. However, analytic flexibility only translates into elevated false positive rates when combined with selective analysis reporting. If researchers reported the results of all analysis pipelines used in their studies, then it would not be problematic.

To the author’s knowledge, there is no evidence that fMRI researchers actually engage in selective analysis reporting. But researchers in other fields do appear to pursue this strategy.

In my experience, fMRI researchers are actually fairly conservative in terms of using different analyses, and certainly I doubt anyone has ever run thousands of them just to get the result they want and I'd estimate that most published findings are not the result of more than a handful of 'attempts' at most.

However it's a serious concern that it could happen, and importantly it's getting ever-easier to do this, with the continuing increase in computer power making running an analysis quicker and cheaper than ever. As to what to do about it, Carp makes several suggestions, and here's one I made earlier...

Joshua Carp (2012). On the plurality of (methodological) worlds: estimating the analytic flexibility of fMRI experiments Front. Neurosci. DOI: 10.3389/fnins.2012.00149

Getting The Position Right For EEG

In science, it's often the most 'boring', easily overlooked factors that determine whether an experiment succeeds or fails.

A new paper reveals strong effects of body posture on brain electrical activity: Subject position affects EEG magnitudes. Just lying face-up as opposed to face-down can powerfully affect the signal measured using electroencephalography (EEG), according to Justin Rice and colleagues of New York.

Here's why: EEG uses electrodes, placed on the scalp, to measure the electrical potentials produced by brain firing.

The signal recorded depends, however, not just on the brain activity but also on the quality of the electrical conduction between the brain and the scalp: the signals have to travel through the fluids surrounding the brain, then the skull, and finally the skin, before they're detected.

EEG users sometimes think of brain-scalp conductivity as a fixed factor, that they can't control and don't have to worry about. However, Rice et al point out that the position of the brain shifts within the skull depending upon your posture.

This is because there's a bit of extra room in there, leaving space for the brain to "bounce around" a little within its fluid cavity. If you're lying on your back, the brain will lie closer to the back of the skull; if you're on your front, it'll be further forward, and so on. So the fluid layer between brain and skull will be corresponding thinner, or thicker.

In a healthy brain the change is only about 1 mm, but the fluid layer's only 3 mm at most, so that's a big change.

Others have recognized this problem before, but Rice et al's data are the clearest evidence yet that posture is a major factor. They showed that subjects lying on their back (supine) showed significantly stronger activity over the back of the brain - which makes sense, as it brings the brain closer to the electrodes. Lying face down (prone) made activity weaker and sitting was in between.

Interestingly - and worryingly - the effect was stronger depending upon the kind of activity being measured. For most kinds of brain activity it was about 40% higher but for gamma waves - the hottest thing in EEG right now - it was almost 80%.

So gamma band activity is especially sensitive to posture, and that raises the worrying possibility that even slight differences in head position between individuals could account for 'differences' in gamma power recorded, for example in studies comparing neurological patients and healthy controls; if the controls are sitting up straight while the patients are slouching back... the patients would seem to have more gamma.

Rice JK, Rorden C, Little JS, and Parra LC (2012). Subject position affects EEG magnitudes. NeuroImage PMID: 23006805

Brain Wiring - More Mess Than Manhattan?

Earlier this year, Harvard neuroscientist Van J. Wedeen and colleagues published a Science paper saying that brain white matter 'wiring' is organized in a grid-like fashion, with sheets of fibres crossing each other.

As Ed Yong put it, that the brain is full of Manhattan-like grids.

However, they were wrong - and that neat grid structure was purely an artefact of the method they used. So say London-based critics Marco Catani and colleagues in a Technical Comment just published.

Catani et al argue that the analysis Wedeen et al used was unable to distinguish two crossing fibres, unless the angle between them was very large, i.e. close to a right angle. In other words, they only saw right angles - and hence neat parallel sheets - but the other angles were still there.
 
They present some data of their own, showing the distribution of fibre-crossing angles in 10 healthy brains:

This shows no clear peak at 90 degrees as the grid theory proposes. Some fibres do cross at right angles, but only about 12% of them. The rest cross at a wide range of other angles.

Catani et al finish up by saying that we know, from cutting up brains, that it's just not a grid:

Crossing ... fanning, merging, and kissing are other modalities that are frequently observed in postmortem anatomy and not visible with current diffusion methods. Finally, the grid model does not take into account the presence of thalamic fibers, which project radially in all brain regions. This implies that most white matter voxels have multiple populations of fibers... merging at progressively tangential angles to reach the same cortical areas. Current tractography reconstructions are biased toward solving only crossing. This information is well known to anatomists, and there is a serious risk in proposing the grid model as “a means to validate MRI tractography through consistency with grid structure”. 

Ouch!

Wedeen et al respond robustly, however. They defend the accuracy of their technique, holding that it's more sensitive than alternatives, and say that even supposing their method had a bias towards detecting right angles, that wouldn't explain why they saw neat overlapping sheets.

They also point out that some earlier neuroanatomists did spot the kind of abrupt 'corners' and orthogonal tracts predicted by their theory, such as this drawing of a monkey brain:

They conclude:

The thesis that brain pathways adhere to a simple geometric system best accounts for the available evidence—not like London, but Manhattan; not unfathomable, but unlimited.

Marco Catani, Istvan Bodi, and Flavio Dell’Acqua (2012). Comment on “The Geometric Structure of the Brain Fiber Pathways” Science DOI: 10.1126/science.1223425

When Data Filtering Introduces Bias (fMRI Edition)

A couple of months ago I blogged about a paper showing that 'filtering' of EEG data can create spurious effects.

Now, we read about another form of bias that filters can introduce, this time for fMRI: Filtering induces correlation in fMRI resting state data.

Australian neuroscientists Catherine Davey and colleagues consider temporal filtering of fMRI data in studies looking at correlation (brain functional connectivity).

Because both very high frequency and very slow changes in the fMRI signal are probably caused by artefacts, rather than interesting brain signals, it's common to use a filter to try and extract the medium-frequency changes that are of most interest (e.g. approximately 0.01 to 0.1 Hz).

However, while this filtering is very useful, Davey et al show that it can - ironically - create artefacts of its own: here's the data from one volunteer scanned during a simple task and then analyzed in 4 different ways:

Without filtering (A) there's a huge amount of 'connectivity' - too much to be realistic. This is why filtering is important.

But filtering, without correcting for the effects of the filter, actually makes things worse (B). It solves one problem but at the cost of creating another. The problem is those pesky autocorrelations. The authors say, however, that they've calculated a way to correct for filter-induced correlations (D) and that this gives more realistic results. They recommend that this should be used in future connectivity studies, but don't go into much detail regarding the question of what this means for the existing literature.

Perhaps data 'filtering' is a misleading term. It implies that all you're doing is removing the unwanted noise, leaving pristine, crystal clear data, a bit like a water filter. Mmm. What could go wrong? In fact mathematical 'filters' can put stuff into the data as well as take it out, so should we stop using that word and just call them what they are: modifications?

Davey CE, Grayden DB, Egan GF, and Johnston LA (2012). Filtering induces correlation in fMRI resting state data. NeuroImage PMID: 22939874

Naomi Wolf's "Vagina"

Naomi Wolf's "Vagina" is full of bad science about the brain - is an article I wrote for the New Statesman. It's about a new book which is... not very good.

I didn't come up with the title by the way, but I do rather like it.

See also the Neurocritic's take.

Brains In Motion Are Bad For Neuroscience

A new paper in Human Brain Mapping reports on: Functional magnetic resonance imaging movers and shakers: Does subject-movement cause sampling bias?

Head movement is a well known problem that can badly impact the quality of neuroimaging data, introducing spurious signals and obscuring real ones. It's an issue for all brain scanning research but according to Wylie and colleagues, authors of this paper, it's especially serious for studies comparing disease patients to healthy controls.

The authors got 34 people with multiple sclerosis (MS) to perform some simple cognitive tasks during fMRI scanning. They found that the harder the task was, the more the patients moved around during the scan; and in those with more severe MS, the correlation between difficulty and motion was even stronger. In healthy people, harder tasks only caused slightly more motion.

And the more people moved, the less brain activity was recorded, probably because movement degraded the data quality.

fMRI data associated with severe head movement is frequently discarded. In discarding these data, it is often assumed that head-movement is a source of random error, and that data can be discarded from subjects with severe movement without biasing the sample. We tested this assumption by examining whether head movement was related to task difficulty and cognitive status among persons with multiple sclerosis (MS).

[In the MS patients] there was a linear increase in movement as task difficulty increased that was larger among subjects with lower cognitive ability. Analyses of the signal-to-noise ratio (SNR) confirmed that increases in movement degraded data quality. Similar, though far smaller, effects were found in healthy control subjects. Therefore, discarding data with severe movement artifact may bias multiple sclerosis samples such that only those with less-severe cognitive impairment are included in the analyses. However, even if such data are not discarded outright, subjects who move more will contribute less to the group-level results because of degraded SNR.

Oh dear. fMRI researchers use two main ways to deal with motion - correction, and rejection. Either you try to take account of movement and analyze the data, or you just throw out the results from people who move a lot. Most people use a combined approach, chucking out the really heavy movers and then using correction on the rest.

This paper however shows that both techniques have problems. Despite motion correction, data from heavy movers has a lower signal to noise ratio, so if you include them, they will "dilute" your sample. However, if you chuck them, that'll also introduce bias, because heavy movement is not random - people with more severe MS move more, so by excluding heavy movers, you'll be excluding severe cases.

Informally, every neuroimaging researcher knows that some people move more than others. Learning to spot likely "movers" and avoid wasting money on scanning them is a fine art. In my experience just about every "patient" population move, on average, more than healthy controls, and children and the elderly move more than young adults.

I'm not sure there's an ideal solution but perhaps the best approach is to run all analyses (at least) twice, once including everyone, regardless of movement, and then again, with strict movement exclusion criteria. Results consistent across both analyses are probably solid.

Wylie GR, Genova H, Deluca J, Chiaravalloti N, and Sumowski JF (2012). Functional magnetic resonance imaging movers and shakers: Does subject-movement cause sampling bias? Human brain mapping PMID: 22847906

A Case Study in Voodoo Genetics

A new review of published studies looking at the relationship between a gene and brain structure offers a sobering lesson in how science goes wrong.

Dutch neuroscientists Marc Molendijk and colleagues took all of the studies that compared a particular variant, BDNF val66met, and the volume of the human hippocampus. It's a long story, but there are various biological reasons that these two things might be correlated.

It turns out that the first published reports found large genetic effects, but that ever since then, the size of the effects has dropped, with the latest studies finding no effects at all -

A cumulative meta-analysis confirms that as more studies on BDNF val66met have appeared, the overall effect estimate has steadily declined -

Finally, the authors found signs of publication bias: there were three small, imprecise studies that reported very large effects of the gene, but no such studies finding no effect (or a reverse effect). You'd expect that small and noisy studies would have a lot of random variation so they wouldn't all be positive even if there was a true effect; that all of the published ones were positive, suggests that null findings are out there, unreported.

Overall, this suggests that val66met probably isn't associated with hippocampus volume after all, and that the early studies showing that it was, were misleading. There's no reason to think that the early studies were wrong as such - they may have accurately reported an effect in the small sample of people they looked at, but it was only a chance finding.

I know a lot of neuroscientists who are now fairly skeptical of this whole genre of candidate gene studies; there was much excitement 5 or 10 years ago, but in retrospect, most of these studies were too small, and the publication process meant that it was the random chance findings that were most likely to get published.

However, we need to avoid any sense of complacency. Until we fix the scientific process, the same thing will happen again.

Molendijk ML, Bus BA, Spinhoven P, Kaimatzoglou A, Voshaar RC, Penninx BW, van Ijzendoorn MH, and Elzinga BM (2012). A systematic review and meta-analysis on the association between BDNF val(66) met and hippocampal volume American journal of medical genetics B Neuropsychiatric genetics PMID: 22815222

Brain Scanning... Or Vein Scanning?

Many fMRI studies of brain activity could be biased by the effect of large blood vessels, according to an interesting new report: Origins of intersubject variability of BOLD and arterial spin labeling fMRI.

fMRI measures BOLD, the Blood Oxygenation Level Dependent response. As the name says, BOLD is when a bit of the brain becomes more active, it uses more oxygen, and the oxygenation level of the blood in the area drops - although it then increases to compensate, and it's the increase that most fMRI picks up.

There's a catch though: blood flows. Specifically, it flows from arteries, into tissues - the brain, in this case - and then into veins. Blood leaving the brain tends to end up in the larger veins and, being large, these exert a large effect on BOLD - even though they're some distance from the true site of neural activation.

So, the worry is that BOLD blobs may be shifted towards the nearest large vein, reducing the accuracy of fMRI. It's a well-recognized issue, but it's not clear just how serious it is... or what we can do about it.

Enter Canadian researchers Ismael Gaxiola-Valdez and Bradley Goodyear. By comparing BOLD to the alternative method of ASL - They show that the large vessel effect does happen to BOLD signals, shifting the activation blobs nearer to the surface of the brain (where veins are.) However, the effect is more pronounced when the blobs represent absolute BOLD changes. When blobs represent the z score - which they usually do - the difference is smaller.

That's somewhat reassuring... but here's the bad news. Some peoples' brains show larger BOLD changes than others. Gaxiola-Valdez and Goodyear show that these individual differences in BOLD magnitude are probably driven by large vessel effects because as the pic at the top shows, BOLD was most variable (between people) at the surface, i.e. near the veins, while ASL, less subject to the vein bias, was most variable deep in the brain, as you'd expect.

Worse, the size of people's BOLD and ASL signal changes (to the same task) were not correlated (or only weakly) when the BOLD study used short task blocks. This calls into question short-block or event-related fMRI experiments that measure differences between people or between groups: the differences might be in the veins, not the brain. With longer task blocks, however, there was a good correlation with the ASL.

Overall, it's an important paper and a reminder that, although neuroscientists sometimes treat the brain as separate from the rest of the body, it's not.

Gaxiola-Valdez I, and Goodyear BG (2012). Origins of intersubject variability of blood oxygenation level dependent and arterial spin labeling fMRI: implications for quantification of brain activity. Magnetic resonance imaging PMID: 22795932

Whole Brain Teaching...?

Oh dear. The Kansas City Star asks: Teachers learn ways to keep students' attention, but are brain claims valid?

Probably not. Unless you're buying a brain scanner or a plush brain, product 'brain claims' are generally just marketing patter. But let's see.

When Chris Biffle called out the word "Class!" Wednesday morning at Walsh University, 450 teachers and administrators yelled back, "Yes!"

"Class class?" he said. "Yes! Yes!" they replied.

"Classity classity," he said.
"Yessity yessity," they chanted back.

Biffle, one of the co-founders of Southern California-based Whole Brain Teaching LLC, is leading a two-day conference about his method. He calls the technique "Class-Yes." Whole Brain Teaching's website says "Class Yes" activates the prefrontal cortex of the brain and "readies students for instruction"...

Whole Brain Teaching reminds me of Brain Gym, a notorious bit of British neuro-nonsense from a few years ago. According to the WBT research page, they have over 50,000 registered teachers and 2 million views of their videos. This also informs us that:

Class-Yes: Our primary attention-getter activates the prefrontal cortex, often called the CEO of the brain... Little if any learning can take place if the prefrontal cortex is not engaged.

while even "mirror neurons" have a role to play:

Mirror: Many brain scientists believe that we learn by mirroring the gestures and activities of others. They have identified mirror neurons scattered throughout the brain that are activated by mimicking the behavior we observe. Our own experience in WBT classroom indicates that when a class mirrors our gestures and, when appropriate, repeats our words, a powerful learning bond is created between students and teachers.

There are lots of problems here, but here's the most fundamental: the theory behind the system seems to be that activating particular parts of students' brains, through a special task, will help them to use that part of their brain when it comes to the actual lesson a few minutes later. But I know of no evidence that bits of the brain "warm up" like that; if anything they're more likely to "wear out" through lack of energy and nutrients although I don't think that's likely either.

If such warm-ups did work, your best bet for activating your primary visual cortex, for example, would be to stare at a rapidly-changing pattern of random colors for a few minutes. That wouldn't improve your vision. It would just give you a headache.

In fact, why not just activate your entire noggin, pharmacologically? Just grab some pentylenetetrazol - a drug that blocks inhibitory signals between brain cells. Snort a few lines of that and if you survive the resulting seizure, go and learn something and see if you're really good at it.

I'm not saying Whole Brain Teaching is useless, I'm not saying anything about the method itself, but the "brain" claims are misleading. Many of the things they recommend are teaching aids and classroom exercises, and no doubt those are helpful. Plus, psychological factors like teacher motivation, student engagement, and a positive atmosphere are vital in learning, and it doesn't matter if you achieve them through neurosciencey gimmicks, they're still going to help... well, except in terms of educating people to spot neurosciencey gimmicks.

But that's teaching. It's nothing to do with the brain.

When Data Filtering Introduces Bias

Oh no. Another worrying methods problem for neuroscience, this time for electrophysiologists: Systematic biases in early ERP and ERF components as a result of high-pass filtering.

The event-related potential (ERP) and event-related field (ERF) techniques provide valuable insights into the time course of processes in the brain. Researchers commonly filter the data to increase the signal-to-noise ratio. However, filtering may distort the data, leading to false results. Using our own EEG data, we show that acausal high-pass filtering can generate a systematic bias easily leading to misinterpretations of neural activity... among 185 relevant ERP/ERF publications, 80 used cutoffs above 0.1Hz. As a consequence, part of the ERP/ERF literature may need to be re-analyzed.

The problem in brief: many researchers use a high-pass filter on their electroencephalography (EEG) and magnetoencephalography (MEG) recordings of brain electrical activity. A high-pass filter removes low frequency (i.e. slow) changes from the signal. These slow fluctuations are often considered to be mere "noise".

The problem is that these filters have side effects: as well as 'cleaning up' the data, they can also distort it. There are two main kinds of filter: causal filters are well-known to mutate the signal. Acausal high-pass filters avoid these dramatic artefacts -

But David Acunzo and colleagues point out that acausal filters can actually be more dangerous, because they still distort the data, just in more subtle ways that are harder to spot. In particular, acausal filters can alter the signal at time points before the true signal begins. See the pic above.

That's not necessarily a problem in all cases, but it's certainly bad news for researchers interested in measuring exactly when neural responses happen.

The authors highlight an area of neuroscience where this problem could be misleading researchers. The very earliest brain responses to visual stimuli, about 90 milliseconds after the stimulus onset, is called the "C1" response. Classically, it was thought that the size of the C1 wave was purely a 'bottom-up' phenomenon, determined only by the brightness etc. of the stimulus. But recently, studies have reported 'top down' modulation of C1 by attention, emotional state, etc.

Acunzo et al point out that many of these studies used strong acausal filtering and that what might be happening is that attention actually causes late changes to the visual response, but that due to filtering artefacts, these late changes appear in the data sooner than they really happen. They advise that only weak (low threshold) high-pass filters should be used, and that interesting findings in filtered signals need to be checked against the raw data.

Acunzo DJ, Mackenzie G, and van Rossum MC (2012). Systematic biases in early ERP and ERF components as a result of high-pass filtering. Journal of neuroscience methods PMID: 22743800

False Positive Neuroscience?

Recently, psychologists Joseph Simmons, Leif Nelson and Uri Simonsohn made waves when they published a provocative article called False-Positive Psychology

The paper's subtitle was "Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant". It explained how there are so many possible ways to gather and analyze the results of a (very simple) psychology experiment that even if there's nothing interesting really happening, it'll be possible to find some "significant" positive results purely by chance. Then you could publish those 'findings' and not mention all the other things you tried.

It's not a new argument, and the problem has been recognized for a long time as the "file drawer problem", "p-value fishing", "outcome reporting bias", and by many other names. But not much has been done to prevent it.

The problem's not just seen in psychology however, and I'm concerned that it's especially dangerous in modern neuroimaging research.

*

Let's assume a very simple fMRI experiment. The task is a facial emotion visual response. Volunteers are shown 30 second blocks of Neutral, Fearful and Happy faces during a standard functional EPI scanning. We also collect a standard structural MRI as required to analyze that data.

This is a minimalist study. Most imaging projects have more than one task, commonly two or three and maybe up to half a dozen, as part of one scan. If one task failed to show positive results, it need never be reported at all, so additional tasks would compound the problems here.

Our study is comparing two groups: people with depression, and healthy controls.

How many different ways could you analyze that data? How much flexibility is there?

*

First some ground rules. We'll stick to validated, optimal approaches. There are plenty of commonly used less favoured approaches, like using uncorrected thresholds (and then, which ones?) or voodoo stats, but let's assume we want to stick to 'best practice'.

As far as I can see, here's all the different things you could try. Please suggest more in the comments if you think I've missed any:

First off, general points:

• What's the sample size? Unless it's fixed in advance, data peeking - checking whether you've got a significant result after each scan, and stopping the study when you get one - gives you multiple bites at the cherry.
• Do you use parametric, or nonparametric analysis?

Now, what do you do with the data?

• Preprocessing
• How much smoothing?
• Do you reject subjects for "too much head movement"?  If so, what's too much?
• Straightforward whole-brain general linear model (GLM) analysis followed by a group comparison.
• What's the contrast of interest? You could make a case for Fear vs Neutral, Happy vs Neutral, Happy vs Fear, "Emotional" vs Neutral.
• Fixed effects or random effects group comparison?
• Do you reject outliers? If so, what's an 'outlier'?
• Do you consider all of the Fear, Happy and Neutral blocks to be equivalent, or do you model the first of each kind of block seperately? etc.
• "Region of Interest" (ROI) GLM analysis. Same options as above, plus:
• Which ROI(s)?
• How do you define a given ROI?
• Functional connectivity analysis
• Whole-brain analysis, or seed region analysis?
• If seed region, which region(s)?
•  Functional connectivity in response to which stimuli?
• Dynamic Causal Modelling?
• Lots and lots of options here.
• MVPA?
• Lots and lots of options here.

But remember we also got structural MRIs, and while they may have been intended to help analyze the functional data, you could also examine structural differences between groups. What method?

• Manual measurement of volume of certain regions.
• Which region(s)?
• VBM.
• Cortical morphometry.
• What measure? Thickness? Curvature...?

That's just the imaging data. You've almost certainly got some other data on these people as well, if only age and gender but maybe depression questionnaire scores, genetics, cognitive test performance...

• You could try and correlate every variable with every imaging measure discussed above. Plus:
• Do you only look for correlations in areas where there's a significant group difference (which would increase your chances of finding a correlation in those areas, as there'd be fewer multiple comparisons)?
• You could define subgroups based on these variables.

*

So even a very straightforward experiment could give rise to hundreds or thousands of possible analyses. 1 in 20 of these would give a statistically significant result at p=0.05 by chance alone, and even if you throw out half those for being "in the wrong direction" (and that's subjective in most cases) you've got plenty of false positives.

This problem is growing. As computation power continues to expand, running multiple analyses is cheaper and faster than ever, and new methods continue to be invented (DCM and MVPA were very rarely used even 5 years ago.)

I want to emphasize that I am not saying that all fMRI studies of this kind are in fact junk. My worry is that it's hard to be confident that any given published study is sound, given that papers are written only after all the data has been collected and analyzed.

I'll also point out that some imaging research, especially what might be called "pure" neuroscience investigating brain function per se rather than "clinical" studies looking at differences between groups, has many fewer variables to play with, but still quite a lot.

As to how to solve this problem, the one solution I believe would work in practice is to require pre-approval of study protocols.

Simmons JP, Nelson LD, and Simonsohn U (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological science, 22 (11), 1359-66 PMID: 22006061