Originally called "Voodoo Correlations in Social Neuroscience", it ended up with the less snappy name of Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. I prefer the old title.
The error in question is now known variously as the "circular analysis problem", "non-independence problem" or "double-dipping" although I still call it the "voodoo problem". In a nutshell it arises whenever you take a large set of data, search for data points which are statistically significantly different from some baseline (null hypothesis), and then go on to perform further statistics only on those significant data points.
The problem is that when you picked out the statistically significant observations, you selected the data points that were especially "good", so if you then do some more analyses only on those data, you are almost guaranteed to find something "good". To avoid this you need to make sure that your second analysis is truly independent of your first one.
Anyway, Vul and Pashler, the main authors of the original voodoo article, have just written a short piece in NeuroImage offering some reflections on the paper and the aftermath. They don't make any major new arguments but it's a good read. Particularly fun is their explanation of what inspired them to look into the voodoo problem:
In early 2005 a speaker in our department reported that BOLD activity in a small region of the brain can account for the great majority of the variance in speed with which subjects walk out of the experiment several hours later (this finding was never published as far as we know). The implications of this result struck us as puzzling, to say the least: Are walking speeds really so reliable that most of their variability can be predicted? Does a focal cortical region determine walking speeds? Are walking speeds largely predetermined hours in advance? These implications all struck us as far-fetched...But they reveal that it was one paper in particular that set them off voodoo-hunting
Our interest in probing the matter was further whetted by an episode occurring a short while later: Grill-Spector et al. (2006) reported that individual voxels in face selective regions have a variety of stable stimulus preferences; in a critical commentary, Baker et al. (2007) found that the analysis used to ascertain this fact implicitly built these conclusions into the method, such that the same analysis applied to noise data (voxels from the nasal cavity) revealed a similar variety of stable preferences. It occurred to us that a similar circularity might underlie the puzzlingly high correlations.To their credit, Grill-Spector et al quickly accepted Baker et al's criticism and admitted that some of their original conclusions had been wrong.
Vul, E., and Pashler, H. (2012). Voodoo and circularity errors NeuroImage DOI: 10.1016/j.neuroimage.2012.01.027
9 comments:
Hi! Nothing to do with your blog posting, but yesterday over on facebook the Guardian newspaper were looking for new science blog writers. I thought of this place!! You may or may not be interested?. G :)
May be somewhat related is an opinion article against small studies (according to several meanings of the term)
in the New York Times from Marco Bertamini and Marcus R. Munafò, psychologists at the University of Liverpool and the University of Bristol, respectively.
http://www.nytimes.com/2012/01/29/opinion/sunday/the-perils-of-bite-size-science.html?_r=2&scp=1&sq=psychological%20science&st=cse
I cite:
///The rise of bite-size science is worrisome. We urge that editors demand more replication of unexpected findings and that the importance that the academic community gives to quantity of citations be balanced with a greater awareness of potential publication bias.
Until then, bite-size science will be hard to swallow.///
Amazing. I had no idea they did this. I'm a complete amateur and I would never make a mistake like this. I guess a lot of it was blindly copying methods.
And please remember that neuroscientists bear the heavy responsibility of informing the actions of the French psychoanalysts according to a judiciary decision,
if a French journalist is to be believed:
http://www.rue89.com/2012/01/26/autisme-le-documentaire-le-mur-condamne-par-la-justice-228785
I cite from that article from a reputable journal:
///« La description de nous-mêmes comme mécanisme biologique se complexifie des hypothèses amenées par les neurosciences, par la biologie fondamentale, par tel chercheur, telle équipe, etc., nous les commentons [...] ».(Said Eric Laurent one of the psychoanalyst who obtained the condemnation of the film maker)
Le tribunal estime donc qu'« il ne peut être retenu de ces propos que le demandeur serait dans le refus des connaissances scientifiques actuelles ».///
My tentative translation of this citation would be:
///" The description of ourselves as a biological mechanism is complexities by hypothesis from neuroscience, fundamental biology, this researcher, this team, etc.. we comment them(...)" (Said Eric Laurent one of the psychoanalyst who obtained the condemnation of the film maker)
The tribunal esteems then that ."it cannot be retain that the psychoanalyst who said that could be in the refusal of the sciences of our times."///
NB: In 10 11 Neuroskeptic posted on how French psychoanalysts use neuroscience.
http://neuroskeptic.blogspot.com/2011/10/le-pack-it-in.htmlOf
Neuroskeptic was discussing the neurospychoanalysis efforts produced by the academic child psychiatrist from the northern French city where the condemnation of the film maker took place- pr Pierre Delion who packs autistic children in cold wet sheets ignoring that deep pressure with dry weighted blankets has a calming effect on many autistic children and adults.
Delion P (2011). Towards a dialogue between psychoanalysis and neuroscience: Connections that are both possible and necessary. Journal of physiology, Paris PMID: 21963531
disturbing.
GCM: Oh, thanks for the tipoff, I missed that!
Interesting and disturbing, considering the implications for who knows how much extant research and claims. One thing bothering me a lot: why didn't someone come up with/was widely known, about this issue already? Really, to be "shaken" by foundational revelations in a paper appearing in 2009? Now that's scary, as scary as zombies.
"Fine minds make find distinctions."
Although I agree 100% that people are using inappropriate statistical techniques in analyzing fMRI data, I wish you would stop calling "voodoo". Voodoo (more properly Vodou) is a religion of the Caribbean, so calling them "voodoo correlations" makes about as much sense as calling them "Christian correlations" or "Hindu correlations".
Now I agree that all religions are stupid, and some religions are stupider than others, I fail to see how labeling statistical errors as voodoo makes any sense though. fMRI errors don't come from any incantations to a supreme invisible man in the sky, they come from using statistics when you don't know how to use them properly. "Double-dipping" is descriptive of the underlying error, while "bullshit" is a perfectly reasonable description of the result of this error. Why drag voodoo into this?
Anonymous Coward but very kind indeed
since you refused to consider that some people like to publish whatever the ethical cost of their frauds from "cleaning the flies" to ignoring what they should know about their trade.
Stats people are-as a specie- very friendly and helpful academic colleagues present in every university.
Post a Comment