More on False Positive Neuroimaging

Back in June, I warned that the ever-increasing number of clever methods for analyzing brain imaging data could be a double-edged sword:

Recently, psychologists Joseph Simmons, Leif Nelson and Uri Simonsohn made waves when they published a provocative article called False-Positive Psychology - Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant.

It explained how there are so many possible ways to gather and analyze the results of a  simple psychology experiment that, even if there's nothing interesting really happening, it'll be possible to find some "significant" positive results purely by chance...

The problem's not just seen in psychology however, and I'm concerned that it's especially dangerous in modern neuroimaging research.

In a comment on that post, The Neurocritic pointed out that Michigan PhD student Joshua Carp had put forward the same argument in a conference presentation, several months previously.

Now Carp's published a paper on the topic: On the plurality of (methodological) worlds: estimating the analytic flexibility of fMRI experiments. It's free to access, so check it out.

Whereas I just talked the talk by listing lots of possible ways in which you could analyze a given set of data, Carp walked the walk, and actually did loads of analyses. He took a single dataset, the results of a simple experiment and looked at it in almost 7000 different ways. Each set of results was then thresholded to correct for multiple comparisons in 5 ways, for a grand total of 35,000 outputs.

The variants he considered ranged from how much smoothing to apply, to how to correct for head motion, and many more.

What happened? In a nutshell, the different options made a difference - and the variability was the largest in parts of the brain that were most activated (the "blobs" that lit up). In other words, analytic flexibility makes the most difference in the most interesting places. See the picture at the top.

The location of the maximum peak activation also varied. This is not unexpected, and not, in itself, that worrying - the great majority of the peaks clustered in a few small areas. However, it underlines that different options really can make a difference.

Carp concludes:

Nearly every voxel in the brain showed significant activation under at least one analysis pipeline. In other words, a sufficiently persistent researcher determined to find significant activation in virtually any brain region is quite likely to succeed...

If investigators apply several analysis pipelines to an experiment, and only report the analyses that support their hypotheses, then the prevalence of false positive results in the literature may far exceed the nominal rate. However, analytic flexibility only translates into elevated false positive rates when combined with selective analysis reporting. If researchers reported the results of all analysis pipelines used in their studies, then it would not be problematic.

To the author’s knowledge, there is no evidence that fMRI researchers actually engage in selective analysis reporting. But researchers in other fields do appear to pursue this strategy.

In my experience, fMRI researchers are actually fairly conservative in terms of using different analyses, and certainly I doubt anyone has ever run thousands of them just to get the result they want and I'd estimate that most published findings are not the result of more than a handful of 'attempts' at most.

However it's a serious concern that it could happen, and importantly it's getting ever-easier to do this, with the continuing increase in computer power making running an analysis quicker and cheaper than ever. As to what to do about it, Carp makes several suggestions, and here's one I made earlier...

Joshua Carp (2012). On the plurality of (methodological) worlds: estimating the analytic flexibility of fMRI experiments Front. Neurosci. DOI: 10.3389/fnins.2012.00149

A New Theory of Psychosis?

A team of British neuroscientists led by the (in)famous David Nutt says that magic mushrooms offer a new theory of psychosis: Functional Connectivity Measures After Psilocybin Inform a Novel Hypothesis of Early Psychosis

It's a reanalysis of a study from earlier this year, which got quite a lot of attention, in which 15 volunteers were injected with psilocybin - the major active hallucinogenic ingredient in 'magic mushrooms' - during an fMRI scan.

In a nutshell, the rather interesting proposal in the new paper is that psilocybin may cause mind-altering effects by blurring the difference between the brain networks responsible for 'internal' and 'external' thought.

Activity in the internal "default mode network" (DMN) is generally anti-correlated with the "task-positive network" (TPN) - when one is higher, the other's lower. The DMN is active when you're not doing much - hence 'default' while the TPN comes online when you're engaged in a particular mental activity.

Nutt's team say, however, that their functional connectivity fMRI data show that after psilocybin, activity in these two networks becomes positively correlated - an unusual pattern. They write:

Increased DMN-TPN coupling has been found in people at high risk of psychosis and an inability to distinguish between one’s internal world and the external environment, sometimes referred to as “disturbed ego boundaries,” is a hallmark of early psychoses and the psychedelic state.

One of our volunteers reported the following after psilocybin: “It was quite difficult at times to know where I ended and where I melted into everything around me.”

To be honest I'd need to see a replication before I put too much faith in this, because this kind of post-hoc reanalysis of fMRI data is very flexible and therefore prone to false positives, but it's an interesting idea, and at least it provides a clear theory for further research.

Carhart-Harris RL, Leech R, Erritzoe D, Williams TM, Stone JM, Evans J, Sharp DJ, Feilding A, Wise RG, and Nutt DJ (2012). Functional Connectivity Measures After Psilocybin Inform a Novel Hypothesis of Early Psychosis. Schizophrenia bulletin PMID: 23044373

Getting The Position Right For EEG

In science, it's often the most 'boring', easily overlooked factors that determine whether an experiment succeeds or fails.

A new paper reveals strong effects of body posture on brain electrical activity: Subject position affects EEG magnitudes. Just lying face-up as opposed to face-down can powerfully affect the signal measured using electroencephalography (EEG), according to Justin Rice and colleagues of New York.

Here's why: EEG uses electrodes, placed on the scalp, to measure the electrical potentials produced by brain firing.

The signal recorded depends, however, not just on the brain activity but also on the quality of the electrical conduction between the brain and the scalp: the signals have to travel through the fluids surrounding the brain, then the skull, and finally the skin, before they're detected.

EEG users sometimes think of brain-scalp conductivity as a fixed factor, that they can't control and don't have to worry about. However, Rice et al point out that the position of the brain shifts within the skull depending upon your posture.

This is because there's a bit of extra room in there, leaving space for the brain to "bounce around" a little within its fluid cavity. If you're lying on your back, the brain will lie closer to the back of the skull; if you're on your front, it'll be further forward, and so on. So the fluid layer between brain and skull will be corresponding thinner, or thicker.

In a healthy brain the change is only about 1 mm, but the fluid layer's only 3 mm at most, so that's a big change.

Others have recognized this problem before, but Rice et al's data are the clearest evidence yet that posture is a major factor. They showed that subjects lying on their back (supine) showed significantly stronger activity over the back of the brain - which makes sense, as it brings the brain closer to the electrodes. Lying face down (prone) made activity weaker and sitting was in between.

Interestingly - and worryingly - the effect was stronger depending upon the kind of activity being measured. For most kinds of brain activity it was about 40% higher but for gamma waves - the hottest thing in EEG right now - it was almost 80%.

So gamma band activity is especially sensitive to posture, and that raises the worrying possibility that even slight differences in head position between individuals could account for 'differences' in gamma power recorded, for example in studies comparing neurological patients and healthy controls; if the controls are sitting up straight while the patients are slouching back... the patients would seem to have more gamma.

Rice JK, Rorden C, Little JS, and Parra LC (2012). Subject position affects EEG magnitudes. NeuroImage PMID: 23006805

Are Gay Men Happier?

A neat little study from UCLA psychologists Francisco J. Sánchez and colleagues examines the mental health of homosexual men using a unique identical twin design.

The paper kicks off with a remarkably lucid introduction:

Men would rather drive around lost than stop and ask for directions. Although this is a gross stereotype, the notion that men should be self-sufficient and able to solve their own problems is a dominant ideal within traditional views of masculinity... men who rigidly adhere to such ideals may harm their own health if they avoid seeking help when they need it.

In general it reads more like a blog post than an academic paper, which is great. If all papers (and especially social science ones) were written this way, I would be a much happier man. Speaking of happy men...

The authors' basic idea is that many men, wishing to appear 'manly', don't talk about or get help for their problems, especially psychological issues: boys don't cry, and men certainly don't. However, the authors argue that gay men, generally less encumbered by traditional masculinity, may be an exception to this rule.

So they took 38 pairs of male 'identical' twins, who grew up together, but who weren't quite identical: one of each was gay, and one straight. By controlling for most genetic and environmental factors, these twins provide a kind of natural experimental test of the effects of homosexuality per se. Not a perfect one, but about as good as we're going to get.

In accordance with the authors' predictions, gay men were indeed more open to seeking psychological help.

But unexpectedly, they were actually less likely to report experiencing psychological distress (on this scale). That's surprising, given several previous reports of higher rates of mental illness in homosexuals, which has been dubbed 'velvet rage'.

Sánchez et al's data suggest that gay men may be, er, more gay (...the other kind), and that their increased rates of diagnosed mental illness are a product of their greater willingness to seek help: maybe straight men are just in denial.

But there's a lot of caveats here. It's a small study, based purely on self report measures, and the gay twins were compared to their own straight twins, but those twins are quite possibly not typical of straight men in general. It might also be that having a straight twin makes life easier for gay men. Still, it's an interesting set of data.

Sánchez FJ, Bocklandt S, and Vilain E (2012). The Relationship Between Help-Seeking Attitudes and Masculine Norms Among Monozygotic Male Twins Discordant for Sexual Orientation. Health Psychology PMID: 23025300

The Two Problems With Science

There's lots of concern at the moment over mistakes, misconduct and misbehaviour in science.

This concern is a good thing. There are serious, systemic problems with modern science as I and many others have long argued.

However, I worry that much of the recent discussion has failed to distinguish between two fundamentally distinct problems. On the one hand, we have outright fraud - i.e. making up data, or otherwise lying, breaking the basic rules of science.

On the other hand we have questionable practices such as: publication bias, p-value fishing, the File Drawer, sample size peeking, post-hoc storytelling, and all of the other dark arts that can lead to false positive science. These are permissible, even encouraged, by the current rules of doing and publishing science.

These two problems are similar in some ways - they're both "bad science", they both lead to failures to replicate, etc. - but in underlying essence they're very different, so much so that I'm not sure they can be usefully discussed in the same breath.

Fraud and questionable practices are different in terms of their harms. Fraud is a more serious act and it causes local harm, introducing major errors into the record. But in terms of its overall effects, I believe questionable practices are worse, as they systematically distort science: ensuring that, in some cases, it is difficult to publish anything but errors.

Fraud and questionable practices call for different solutions. Broadly speaking, fraudsters break the rules, so to stop them we need to enforce those rules, via deterrence, detection, and punishment - like with any criminal act. With questionable practices, it's the opposite: here the problem is the rules (or the lack of them), and the solution is to reform the system.

It's been suggested that fraud and questionable practices share a common cause in the "pressure to publish", the "publishing environment", the "culture" of modern science etc. But while this is a good explanation for questionable practices, I don't think this can explain fraud, any more than, say, the desire for money can explain theft.

Yes, thieves desire money, and yes they steal in order to get money, but everyone else wants money as well, yet most of us don't steal, so that's not an explanation. Frauds fake data to produce publications. But all scientists are under pressure to produce good publications and they always have been - which is why fraud is not new - what's changed recently is the criteria for a 'good' publication.

Now in retrospect, I blurred these distinctions somewhat with my own 9 Circles of Scientific Hell, in which I placed 6 questionable practices and 2 forms of misconduct on the same scale of "sinfulness". In fact there are two distinct hierarchies. In my defence though, that was a cartoon.

I think finance offers a great analogy here.

In finance, you have some people who break the rules. Bernie Madoff is the current poster boy for this. Such people harm others by outright criminal acts. But then we have the people who play by the rules, and still cause harm. The global financial crisis was in essence caused by all of the major American banks going all-in on a bet, and losing. Yet no-one broke the rules: the regulations allowed banks to gamble. The problem was not rule-breaking, but the rules (or lack thereof).

Here's the curious thing: the financial crisis did more harm than Madoff's scam, even though what Madoff did - theft by fraud - was more immoral than what the bankers did - gambling unwisely.

That's confusing to our ethical sense and our emotions (who should we feel more angry at? Who's 'worse'?) but it's really no surprise: precisely because what the banks did was above board, everyone did it so the damage was huge. If it had been illegal for banks to gamble all their money at once, individual banks might still have broken that rule, locally, but it's unlikely that the system would have been threatened.

Maybe you can see where I'm going with this: everyone following bad rules is often worse than individuals breaking good rules.

Science has its share of fraud. Hauser, Smeesters, Fujii - they broke good rules against such deceit. They are the Bernie Madoffs of science. But then there's 'questionable practices' like publication bias, p-value fishing, the File Drawer, and all the rest, which are allowed, but which are universally acknowledged to be bad for science. Scientists using these dark arts (and I don't know any who never do) may be the Lehman Brothers of science.