False Positive Neuroscience?

Recently, psychologists Joseph Simmons, Leif Nelson and Uri Simonsohn made waves when they published a provocative article called False-Positive Psychology

The paper's subtitle was "Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant". It explained how there are so many possible ways to gather and analyze the results of a (very simple) psychology experiment that even if there's nothing interesting really happening, it'll be possible to find some "significant" positive results purely by chance. Then you could publish those 'findings' and not mention all the other things you tried.

It's not a new argument, and the problem has been recognized for a long time as the "file drawer problem", "p-value fishing", "outcome reporting bias", and by many other names. But not much has been done to prevent it.

The problem's not just seen in psychology however, and I'm concerned that it's especially dangerous in modern neuroimaging research.

*

Let's assume a very simple fMRI experiment. The task is a facial emotion visual response. Volunteers are shown 30 second blocks of Neutral, Fearful and Happy faces during a standard functional EPI scanning. We also collect a standard structural MRI as required to analyze that data.

This is a minimalist study. Most imaging projects have more than one task, commonly two or three and maybe up to half a dozen, as part of one scan. If one task failed to show positive results, it need never be reported at all, so additional tasks would compound the problems here.

Our study is comparing two groups: people with depression, and healthy controls.

How many different ways could you analyze that data? How much flexibility is there?

*

First some ground rules. We'll stick to validated, optimal approaches. There are plenty of commonly used less favoured approaches, like using uncorrected thresholds (and then, which ones?) or voodoo stats, but let's assume we want to stick to 'best practice'.

As far as I can see, here's all the different things you could try. Please suggest more in the comments if you think I've missed any:

First off, general points:

• What's the sample size? Unless it's fixed in advance, data peeking - checking whether you've got a significant result after each scan, and stopping the study when you get one - gives you multiple bites at the cherry.
• Do you use parametric, or nonparametric analysis?

Now, what do you do with the data?

• Preprocessing
• How much smoothing?
• Do you reject subjects for "too much head movement"?  If so, what's too much?
• Straightforward whole-brain general linear model (GLM) analysis followed by a group comparison.
• What's the contrast of interest? You could make a case for Fear vs Neutral, Happy vs Neutral, Happy vs Fear, "Emotional" vs Neutral.
• Fixed effects or random effects group comparison?
• Do you reject outliers? If so, what's an 'outlier'?
• Do you consider all of the Fear, Happy and Neutral blocks to be equivalent, or do you model the first of each kind of block seperately? etc.
• "Region of Interest" (ROI) GLM analysis. Same options as above, plus:
• Which ROI(s)?
• How do you define a given ROI?
• Functional connectivity analysis
• Whole-brain analysis, or seed region analysis?
• If seed region, which region(s)?
•  Functional connectivity in response to which stimuli?
• Dynamic Causal Modelling?
• Lots and lots of options here.
• MVPA?
• Lots and lots of options here.

But remember we also got structural MRIs, and while they may have been intended to help analyze the functional data, you could also examine structural differences between groups. What method?

• Manual measurement of volume of certain regions.
• Which region(s)?
• VBM.
• Cortical morphometry.
• What measure? Thickness? Curvature...?

That's just the imaging data. You've almost certainly got some other data on these people as well, if only age and gender but maybe depression questionnaire scores, genetics, cognitive test performance...

• You could try and correlate every variable with every imaging measure discussed above. Plus:
• Do you only look for correlations in areas where there's a significant group difference (which would increase your chances of finding a correlation in those areas, as there'd be fewer multiple comparisons)?
• You could define subgroups based on these variables.

*

So even a very straightforward experiment could give rise to hundreds or thousands of possible analyses. 1 in 20 of these would give a statistically significant result at p=0.05 by chance alone, and even if you throw out half those for being "in the wrong direction" (and that's subjective in most cases) you've got plenty of false positives.

This problem is growing. As computation power continues to expand, running multiple analyses is cheaper and faster than ever, and new methods continue to be invented (DCM and MVPA were very rarely used even 5 years ago.)

I want to emphasize that I am not saying that all fMRI studies of this kind are in fact junk. My worry is that it's hard to be confident that any given published study is sound, given that papers are written only after all the data has been collected and analyzed.

I'll also point out that some imaging research, especially what might be called "pure" neuroscience investigating brain function per se rather than "clinical" studies looking at differences between groups, has many fewer variables to play with, but still quite a lot.

As to how to solve this problem, the one solution I believe would work in practice is to require pre-approval of study protocols.

Simmons JP, Nelson LD, and Simonsohn U (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological science, 22 (11), 1359-66 PMID: 22006061

B. F. Skinner vs. the Rorschach Test

What happened when the world's most no-nonsense psychologist took a Rorschach test?

A fun little paper reports on B. F. Skinner's Rorschach results. He agreed to be tested as part of a 1953 project psychoanalysing various eminent scientists. The scientists were anonymous at the time but now Norwegians Cato Grønnerød et al have dug them out of the archives (Skinner has been dead since 1990).

Skinner was the world's leading exponent of behaviourism, a school of thought that held roughly that it's impossible to know anything about "inner" mental states or thoughts, and that they might not even exist, so all we could do was look at and try to predict behaviour (edit: see comments for clarification).

It was never an especially convincing idea to be honest and behaviourism is now pretty much dead although many of the techniques pioneered by Skinner live on in the form of tests on lab animals to determine the addictiveness of drugs and so forth.

But in the mid-20th century it was very popular and Skinner was a well-known figure, the Jonah Lehrer of his day in many ways although rather more controversial.

Anyway. Grønnerød et al report that when Skinner was asked to describe those famous inkblots -

The most evident feature of the protocol is the huge number of responses, showing a highly productive and creative person. But complexity is sacrificed for quantity... No perceptual distortions are evident, and reality testing and ability to function neutrally are in place. We found no signs of cognitive distortions, although some responses have an idiosyncratic twist... He might be an assertive person with a tendency to view relations as generally competitive and an area for the expression of his own needs, rather than an area of mutual support and belonging.

Although he shows an interest in others, the balance between real and whole humans and other human representation suggests that perception of self and others is based more on fantasies and wishes than on real-life perceptions... “Necrotic looking,” “wounded animal,” and “sheep pushing the two wolves away” might reflect projected aggression. These processes point to more primitive defense mechanisms...

Which is exactly the kind of speculation that Skinner spent his career trying to put a stop to. Still, it's an interesting paper, although I think it tells you more about the Rorschach than about Skinner.

Grønnerød C, Overskeid G, and Hartmann E (2012). Under Skinner's Skin: Gauging a Behaviorist From His Rorschach Protocol. Journal of personality assessment PMID: 22731841

Brain Activation Is Pretty Selective

Neuroimaging researchers like to talk about bits of the brain in terms of what kind of stimuli they respond to.

The "Fusiform Face Area (FFA)" and "Parahippocampal Place Area (PPA)" are two of the most popular of these 'clue is in the name' areas. The FFA lights up in response to seeing faces, while the PPA is more into places... so textbooks will tell you.

But how selective are these areas really? We know that the FFA activates more to faces than to other things on average, but is there overlap? Are there non-faces that activate the FFA more than many faces? Or perhaps, are there faces that don't activate the FFA? If there were, it would undermine the whole concept of the FFA as a "face area".

NIH researchers Marieke Mur et al have just examined this question and the results are out now in the Journal of Neuroscience. They show that yes, the FFA is selective for faces and likewise, the PPA is really pretty place-selective.

Which is a nice surprise. The last few weeks have been tough ones for brain scanning, with a crop of studies showing potentially serious flaws in popular neuroimaging methods. I was bracing myself for more bad news when I picked up this paper, but the results are actually quite reassuring.

They fMRI scanned 4 volunteers twice each, and showed them a series of 96 pics - some faces, some places, plus bodies and objects. They found that there was little overlap in the degree of activation across categories: few non-faces activated the FFA as strongly as faces - the closest were bodies, which are living things too so that makes sense. The PPA was even more selective.

See the picture at the top which shows stimuli ranked by how much they activated each area.

Fascinatingly - well, amusingly - the least face-like picture was some garlic, but the most unplaceish image was a cucumber.

Maybe these volunteers just didn't like vegetables.

There was some overlap between faces and non-faces for the FFA activations but it wasn't consistent. The second scan session - with the same 96 pics - found that those non-faces that activated the FFA the first time, didn't do so the second time. So it was just random noise.

Overall this is a lovely study, although it does underline the fact that although these areas are consistently selective, the difference is pretty subtle; there's no "break point". The FFA does prefer faces... but not enormously so.

Mur M, Ruff DA, Bodurka J, De Weerd P, Bandettini PA, and Kriegeskorte N (2012). Categorical, yet graded - single-image activation profiles of human category-selective cortical regions. The Journal of neuroscience : the official journal of the Society for Neuroscience, 32 (25), 8649-62 PMID: 22723705

Starkey vs. Penny - Transcript

The British Twittersphere has been enlivened today by the release of a video showing the 'heated' debate between historian David Starkey and journalist Laurie Penny during a discussion on the English national identity.

Some clips are available on YouTube, but the full video is behind a paywall, so I thought I'd write up the transcript for anyone who's interested.

Personally I think any attempt to define a national (or other) 'identity' can only undermine that identity, just as analyzing a joke renders it unfunny. Identities, like humour, stop making sense when you try to make sense of them; there is nothing rational about them, and to debate them rationally is impossible - this exchange is English in a way that no amount of sensible arguments could be.

Laurie Penny at podium.

LP: “But what’s interesting about the question what is Britishness, is precisely the fact that nobody seems to know. It’s gold for a columnist, it’s gold for any hack, and, you’ve seen 4 people on this panel are probably going to give completely different answers, anybody you ask in this audience is going to give a different answer.

If you ask David Cameron, it’s about tolerance and fairness and inviting French people to take advantage of our lenient tax system; if you ask a pupil at Wellington college, growing up in these lovely surroundings, they’ll tell you, probably about the excellent teaching here, but they’ll also tell you about grime and dubstep and whatever it is they’re listening to; if you ask me, and I’ve been recently been living in America for a while, so, I get asked this question quite a lot, and I bring up things like, the taste of fish and chips on Brighton Beach, and Dr Who, and drinking tea. That’s what it is for me.

If you ask somebody like my colleague Prof. Starkey, it’s playing xenophobia and racial prejudice for laughs, and if you ask people who organize conferences like this, it’s sitting by politely while people play racial prejudice and xenophobia for laughs and pretending that this is an acceptable part of contemporary debate…”

Video cuts – panel discussion

A Man: “We’re setting up a false dichotomy here between Britishness and Englishness; they’re not, at all…”

David Starkey: “There’s an absolute dichotomy. The two are completely different, there’s a historic English identity which goes way back before the concept of Britain. Also this notion that England or Britain as uniquely a nation of immigrants is preposterous - every European country is a blend, of course it is, but the great, there are two great differences. The first…”

Laurie Penny: “I believe you have a house in America, Mr Starkey?”

DS: “I have a house in America.”

LP: “I was wondering, where are you domiciled for tax purposes?”

DS: “I am domiciled here, and I pay full taxes, and can I just say…” (stands up to face audience) “as you have chosen to be personal and invidious, let me share a little story with you. One of the great things that is essential to Britishness is the sense of public duty, that you do things for nothing with organizations that can’t pay. Ms Penny, who has been advertising these great left-wing virtues, and I, were to due to debate for a very impoverished little society called the Thomas Paine Society, on the virtues of a republic on the one hand and a monarchy on the other. I…”

LP: (inaudible)

DS: “I was prepared to do it for free, she insisted on trying to charge such a large fee that the event had to be cancelled. Now I think that that is as mean and grasping as some runt comedian (audience applause), and I will not be lectured to by a jumped-up public schoolgirl like you; I came from the bottom, and I will not have it.”

(applause, laughter, jeers etc.)

(LP takes podium)

LP: “Excuse me, may I respond to that, I think I deserve a right of reply to that. I personally feel that asking to be paid…”

DS: “Did you or did you not do what I just said?”

LP: “I was going to respond to that, but if you…”

DS: “Why not do it straight away. Did you or did you not claim such a large fee that the event had to be cancelled?”

LP: “No, I didn’t, Mr Starkey.”

DS: “Well then, every member of the committee is lying and you, clearly, as usual, are telling the truth.”

LP: “No, this is not what was said to me. I…”

DS: “I have emails. That wonderful… method of tracking…”

Chairman: “Alright, alright, respond to it Laurie…”

LP: “No, actually I don’t think I’m gonna respond to it, one of the… the event was cancelled partly because they couldn’t find a speaker in time, because they were trying to fly me out and part of the reason that… the cost for that on top of my being asked to be paid for two days work, because I actually don’t earn very much…”

DS: “Pffffffffff!”

Chairman: “They had two pretty good speakers there already…”

DS: “No, she backed out because she wasn’t being paid.”

Chairman: “Anyway, we’re running out of time, but I wanna come back to Britishness vs. Englishness…”

LP: “I’m sorry, excuse me - I really don’t appreciate being stood on stage to be personally attacked and having a finger jabbed in my face.”

Video cuts

Chairman: “Laurie?”

LP: “I think I’d like to say something about Britishness in terms of what’s just gone on here, which I feel we’ve moved on from very quickly, I attempted to make a point, and had a person on this panel shouting and pointing in my face, and in fact the point that Professor Starkey raised, about me dropping out of a panel when we were due to speak together, my main reason for doing so was that I didn’t want something like that to happen. I felt like I was being set up to be attacked, and part of the reason I decided to – this is a difficult thing to say and it’s easier to say to event organizers that you’re going to charge a ridiculous sum of money - and be asked to… well not necessarily ridiculous but… and be asked to go.” (cheers / jeers?)

Chairman: “Alright…”

LP: “On the other hand, excuse me, I haven’t finished my point. My point is that this is what debates like this come down to. There’s a civil way of putting… racial prejudice… there’s also…”

(Someone walks in front of camera and makes a ‘cut’ gesture to Chairman)

Chairman: “OK Laurie, I want a final word from…”

LP: “I also want to say that there’s a violence inherent in this discussion and it comes out very, very easily.”

Audience woman: “You started it, that’s the problem, you called him a racist, you know you did…”

LP: “He is a racist.”

Organizer: “OK ladies and gentlemen, we’re going to call a halt at this point…”

This Is Your Brain On Ethics - Really

A new study published in Social Cognitive and Affective Neuroscience offers a look at the neural correlates of ethical decision making: Differential Neural Circuitry and Self-Interest in Real versus Hypothetical Moral Decisions

There have been quite a few studies using neuroimaging to measure activations associated with tackling hypothetical moral dilemmas but what makes the new paper interesting is that the participants were faced with a real moral choice. Well, mostly real.

The task was called "Your Pain, My Gain". The participants were put in the MRI scanner and given £20 at the start of the experiment. Then they could spend some of the money to help save another person from getting electric shocks. The more they spent, the less severe the pain administered to the victim. Video footage of the victim receiving the shocks was then played to the decider, and the process repeated.

Selfish people could choose to keep more of the cash, if they could handle the guilt of knowing that they were shocking their poor victim. At the end of the study, the remaining sum was multiplied by a random factor of between 1 and 10, meaning that there was, potentially, £200 at stake - a serious amount of money.

Actually, the task was a sham - as these experiments usually are. All of the 'shock' videos were pre-recorded, there was no victim, so the decision to keep the money had no effect. But the experimenters tried to make it as believable as possible (out of 18 volunteers, 4 were skeptical, and they were excluded).

There was also a comparison task, which was the same idea, but the participants were told it was purely hypothetical, and would have no real consequences.

The fMRI data showed (although note, some of these did not use multiple comparisons correction) that both the 'real' and hypothetical tasks activated a 'shared moral network':

Although both the hypothetical and the real tasks also led to different patterns of individual activation as well - the authors write:

hypothetical moral decisions mapped closely onto the imagination network, while real moral decisions elicited activity in the bilateral amygdala and anterior cingulate—areas essential for social and affective processes.

Hmm. To be honest the results are a bit messy but the method is extremely interesting. I've written before about the need to keep it real when it comes to stimuli in fMRI experiments so this is important: hypothetical moral dilemmas are no substitute for the real thing. Personally I'm holding out for someone to replicate the Milgram experiment in the MRI scanner although I'm not sure anyone would get ethical approval for that nowadays...

Feldmanhall O, Dalgleish T, Thompson R, Evans D, Schweizer S, and Mobbs D (2012). Differential Neural Circuitry and Self-Interest in Real versus Hypothetical Moral Decisions. Social cognitive and affective neuroscience PMID: 22711879

Beware Stimulus Effects in Psychology

Recently I've blogged about methodological problems in neuroscience research but just to even things out a bit, here's a paper that highlights a potentially serious issue for psychologists - Treating Stimuli as a Random Factor in Social Psychology: A New and Comprehensive Solution to a Pervasive but Largely Ignored Problem

Suppose you want to find out whether people react differently to stimuli from two different groups. The reactions, stimuli, and groups could be anything: maybe you want to see if people prefer listening to sound clips of cats as opposed to dogs. Or maybe you show people photos of blonde men vs dark-haired men and see whether people judge guys with one colour as less trustworthy.

A lot of psychology studies amount to this.

Going with the blonde vs. dark example, suppose you take 1000 volunteers, show them some pictures of blonde guys and dark guys, and get them to rate them on trustworthiness. You find a significant difference between the two groups of stimuli. You conclude that your volunteers are hair-bigots and submit it as a paper. The reviewers think, 1000 volunteers? That's a big sample size. They publish it.

Now that study I just described might be perfectly valid. But it might be seriously flawed. The problem is that while your sample size may be large in terms of volunteers, it might be very small in another way. Suppose you have just 10 photos per group. Your 'sample size', as regards the sample of stimuli, is only 20. And that sample size is just as important as the other one.

It might be that there's no real hair difference in perceived trustworthiness, but there are individual differences - some men just look dodgy and it's nothing to do with hair - and in your stimuli, you've happened to pick some dodgy looking blonde guys. Or whatever.

Now you can run your statistical analyses taking these possible stimulus variation effects into account. But according to Judd, Westfall and Kenny, authors of this paper, this is rarely done. They show with both real and hypothetical data, that unless you take care of this, you can find "statistically significant" differences from pure random noise. This is not a new argument, but they say it's been ignored for too long.

The worst part is that increasing the number of volunteers actually makes it more likely that you'll fall foul of this, not less. Only increasing the stimulus sample size can prevent it.

The paper goes into lots of detail, and tackles various hot potatoes, including one of Daryl Bem's notorious precognition "retroactive priming" experiments. Bem claimed that college students were able to predict the future - they responded differently to different pictures... before the pictures appeared on the screen. The effect was statistically significant and he published it. But Judd et al say that accounting for stimulus variation removes the effect.

Judd CM, Westfall J, and Kenny DA (2012). Treating Stimuli as a Random Factor in Social Psychology: A New and Comprehensive Solution to a Pervasive but Largely Ignored Problem. Journal of Personality and Social Psychology PMID: 22612667

The PR Crisis of Democracy

What will "democracy" mean, for the next generation?

At the moment, Western democracy has a pretty good image. It’s associated with things like: freedom, prosperity, wealth, justice, progress. I'm not saying that democracy actually causes those things; I'm not saying it doesn't. What I mean is that when most people think of democracy, those are what springs to mind, I think. Democracy today has a pretty good image.

Many people assume that democracy will always be attractive, although only a few have been bold enough to say it. But this wasn't always true, and maybe it won't be. 10 years from now, what will people think of when they hear the word “democracy”?

That all depends what happens. Suppose things get worse in:

 

Egypt, Libya: Democracy (at least in theory)… violence, extremists, civil war.

Europe: Democracy… riots, poverty, panic.

America: Democracy… deadlock, paralysis, decline.

I'm not saying democracy really caused those problems or would be responsible if they get worse. Correlation isn't really causation, but psychologically, it feels like it is (that's why we need reminding so often that it isn't.) I'm just summarizing the news headlines from the past couple of years and assuming they continue.

I'm not saying those headlines are justified, either; the media exaggerate, but this is all about perception, and like it or not, people see headlines. And of course there are other old and new democracies around the world that are doing fine - but good news doesn't make the front page.

Put it all together and it doesn’t look so attractive.

This matters. The USSR ended because the people of the USSR looked West and saw a better life. Now there's an assumption that something similar is bound to happen eventually in those countries that are 'still' not democratic. We assume that they will look at democracy and think: me too! For that matter, we assume that most people in a democracy would not vote to end democracy.

But what if democracy loses face?

This hasn't happened yet. It has only just begun, and hopefully can be prevented. Europe today, even Greece today, is surely more attractive than the USSR 1989... but things could get worse, and then where will democracy be?

Brains are Different on Macs

Update - A number of articles linking to this post are wrongly stating that FreeSurfer is medical software used to diagnose diseases or measure the size of brain tumors. It's not. It is purely for research purposes as the software license states, "The Software has been designed for research purposes only and has not been reviewed or approved by the Food and Drug Administration or by any other agency. CLINICAL APPLICATIONS ARE NEITHER RECOMMENDED NOR ADVISED."

Last month, neuroscientists were warned about potential biases in SPM8, a popular software tool for analysis of fMRI data.

Now a paper highlights another software pitfall: The Effects of FreeSurfer Version, Workstation Type, and Macintosh Operating System Version on Anatomical Volume and Cortical Thickness Measurements

FreeSurfer is one of the major image analysis packages and amongst other things, you can use it to measure the size of different parts of the brain.

German Dutch researchers Ed Gronenschild and colleagues took a set of 30 brains and got FreeSurfer to estimate the size and thickness of various structures. Then they did the same thing, on the exact same brains, with a different version of the software.

They found substantial differences in regional volumes, depending upon the version of FreeSurfer used. Running the same version of the software on a Mac vs a PC also created differences, and even the version of Mac OS had an impact.

How much of a difference it made varied by brain location. The differences were 5-15% with version changes. For Mac vs PC and Mac OS updates it was less bad, 2-5% mostly, but in the worst regions - the parahippocampal and entorhinal cortex - it was still almost 15% different. Why those regions are so variable is unclear.

The paper goes into lots more detail, but the lesson for researchers is extremely simple: don't cross the streams of data-analysis. Set up your analysis stream and then use it on all of your data. Same hardware, same software, same settings.

Imagine you're doing a study comparing brain structure in two groups. Halfway through analyzing your data, you upgrade your MacOS. All of the brains you analyze after that will be, say, 5% "bigger". That'll certainly make your data much noisier, and if you happen to analyze most of Group A before Group B, it'll give you a false positive finding.

Sometimes you just can't avoid changes in hardware or software - IT techs have a habit of upgrading things without asking - but in these cases, you should run the same data under the old and the new regime to see if it's making a difference.

Finally, it would be wrong to blame FreeSurfer for this. I'd be surprised if they were any worse than the other software packages. Mixing and matching versions is something that the FreeSurfer developers specifically warn against. This paper shows why.

Gronenschild EH, Habets P, Jacobs HI, Mengelers R, Rozendaal N, van Os J, and Marcelis M (2012). The Effects of FreeSurfer Version, Workstation Type, and Macintosh Operating System Version on Anatomical Volume and Cortical Thickness Measurements. PloS one, 7 (6) PMID: 22675527

Kids Today Are Not Inattentive

There's no evidence that children today are less attentive or more distractible than kids in the past, according to research just published by a team of Pennsylvania psychologists: Long-Term Temporal Stability of Measured Inattention and Impulsivity in Typical and Referred Children.

The study gave a large sample of kids the "Gordon Diagnostic System" GDS test of sustained concentration ability. This dates to the 80s and it consists of a box, with a button, and a display with three digits. There are three different tasks but the main one is a sustained attention test. The goal is to watch a series of numbers and quickly press the button whenever a "1" is followed by a "9". Easy... but it takes concentration to do well.

Over the period of 2000-2006, the researchers gave the GDS to 445 healthy American kids, not diagnosed with any learning or behavioural disorder and not taking medication. They compared their scores to the standardized norms - which were based on a sample of American kids back in 1983.

The results showed that today's kids scored pretty much the same, on average, as the 1983 kids. The average age-standardized scores were extremely close to the 1983 means, across the board. Children diagnosed with ADHD, as expected, scored much worse. Oddly, kids with an Autism Spectrum Disorder did just as badly as the ADHD ones.

One of the researchers on this study is none other than Michael Gordon, who invented the GDS and, one assumes, makes money selling it. (Each GDS kit costs $1595, so someone is making a killing here.) So perhaps we should take this paper with a pinch of salt, because it's kind of an advertisement for the reliability of the GDS.

Still, these results seem pretty solid. That's good news for American children... but bad news for people like Professor Susan Greenfield, who thinks that the internet and videogames are causing an epidemic of ADHD, and all kinds of other problems.

These data suggest rather that, while ADHD diagnoses are certainly rising, children as a whole are not getting less attentive, suggesting that the rise of ADHD is more of a cultural shift.

Mayes, S., Gordon, M., Calhoun, S., and Bixler, E. (2012). Long-Term Temporal Stability of Measured Inattention and Impulsivity in Typical and Referred Children Journal of Attention Disorders DOI: 10.1177/1087054712448961

Big Pharma Read Neuroskeptic?

A few weeks back I blogged about an error-ridden paper from pharmaceutical company Janssen. It  reported on the failure of a candidate antidepressant, JNJ-18038683. But it was rubbish: the Abstract contradicted the results of the study, wrongly claiming that the drug did statistically significantly better than placebo in a particular analysis, when it didn't. And they repeatedly mixed up the names of two other antidepressants.

I posted on May 17th. On May 23rd, a new version of the offending article quietly appeared on the journal's website: here it is. But the old version is still up, although I'm not sure it's intended to be, because the link is gone, so I think the only way to find it is from this blog.

Interestingly, the new manuscript corrects those two issues I noted. As far as I can see those are the only changes. Coincidence?

The original paper was a "Fast Forward" online accelerated publication manuscript, and it's common for these to be corrected in minor ways when they're officially published (i.e. when they appear in print), but that's not what happened here, because the new version is still a Fast Forward. This kind of revision is unusual.

So it seems that someone at Janssen is reading this blog!

If so: Hey. Believe it or not, I'm on your side, I want you to succeed. I suffer from depression, and I've love it if you came up with an actually good new antidepressant. But your industry hasn't released one in at least 10 years (be honest), and you're not fooling anyone with clever statistical tricks.

Forget that, and invest in some proper science. For example, ketamine is clearly the most exciting potential antidepressant right now, and glutamate could be the next serotonin. But because no-one has tested it against an active placebo, no-one, including you, knows whether it's really working. Run a really serious trial of ketamine, if it does work, then that's your next generation of antidepressants right there. If not, better to know that now than later when you've sunk billions into the idea.

Teaching Neuroanatomy With A Showercap

Learning the names and locations of the different parts of the brain is a task that has brought grief to generations of students.

I myself didn't know my caudate from my cingulate cortex all through my undergraduate studies and the first year of my doctorate. I only cracked it after spending a couple of days in the library, surrounded by a stack of anatomy textbooks, copying diagrams and coloring them in over and over until I could do it from memory.

Now a group of Australian physiologists say there's a better way - Showercap Mindmap: a spatial activity for learning physiology terminology and location

Basically, undergraduate students were split into groups of 3 or 4 and each team was given a pack:

The Showercap Mindmap packs included a clear, unmarked plastic shower cap, a whiteboard marker, and 15 sticky, color-coded labels.

Lobe labels were blue (occipital, temporal, parietal, and frontal).

Specialist areas were green (the corpus callosum, Broca’s area, and Wernicke’s area).

Labels relating to information processing were yellow (hearing; heat; pain and temperature; and interpretation and integration of information).

Cortex labels were orange (motor, association, somatosensory, auditory, and visual).

One student on each team wore the cap and the others had 10 minutes to attach the labels to the correct parts of their head, corresponding to the different brain areas, with the help of a neuroanatomy textbook.

This strikes me as a fantastic idea, and something that could actually make learning neuroanatomy fun, or at least a bit more involving than it usually is. The authors of the paper say that students using the method learned more effectively than those using a more conventional approach. Even the cap-wearers benefited. They couldn't see the labels being placed, but they could feel them.

Vanags T, Budimlic M, Herbert E, Montgomery MM, and Vickers T (2012). Showercap Mindmap: a spatial activity for learning physiology terminology and location. Advances in physiology education, 36 (2), 125-30 PMID: 22665427

That Antidepressants In Water Cause Autism Study

Oh dear. The newspapers this morning are reporting that

Autism 'could be triggered by very low doses of anti-depressants or other chemicals found in water supply'

Here's the study. Young fish were exposed to a combination of three drugs, two antidepressants and an epilepsy med, for 18 days.

First off, this study was tiny with an effective sample size of just 6. Three tanks of fish got exposed to the drugs, and three control tanks didn't. There were multiple fish per tank, five in fact, but those are not five independent observations, because they shared a tank. That's just tiny for a drug trial, or any scientific study really.

Next, the drug doses were much higher than in the water supply. Levels of fluoxetine (Prozac) were 700 times higher than observed in drinking water, for carbamazapine it was 400 times higher. And that's based on the authors' figures for drinking water which they admit are "the highest observed concentrations from various systems". The authors defend this by saying that in drinking water there will be other related compounds, on top of the drugs themselves, adding up to a higher dose. OK - but 400, 700 times higher? We've no idea if that's realistic. They don't justify this number.

What did the drugs actually do to the fish? After 18 days of exposure to the drugs, the fish - juvenile fathead minnows - had their brains removed and the expression levels of various genes measured using a genetic microarray.

The drugged minnows had significantly increased expression of a set of 324 genes dubbed "autism genes" ("autism_ideopathic" in the paper.) I'm not going to get into the question of whether these really are autism genes in humans, or whether fish brains are a good of model of humans. Those are hard issues. But what's easy to see is that while this set of genes were apparantly increased, so were many others. It was not specific to 'autism genes'.

The autism genes were upregulated by an average factor of +1.621... but this was only slightly more than the "Parkinson's Disease genes" at +1.56 and the "Multiple Sclerosis" ones at +1.375. Meanwhile, "Bipolar Disorder" genes were down by -1.172. So if antidepressants in the water are causing autism, they're probably also causing (or preventing!) a lot of other problems too.

The authors note that only three of the gene sets were statistically significantly altered, but that doesn't mean those sets were special, this is the fallacy of treating differences in significance levels as evidence of significant differences.

Of 10 more specific "autism gene" sets that they also examined (in the same fish), all were increased by various amounts (+1.050 to +1.537), some of which were significant - but one of those was a set of genes previously reported decreased in autistics, not increased (it was the "synapse" genes from this study).

What these changes in gene expression mean is anyone's guess. Given the small sample size they could be just noise. If not, all they really show is that levels of psychoactive medications that are quite low, but much higher than in drinking water, have affect the brains of fish. We don't know what that effect means, for the fish, let alone humans.

Early life antidepressant exposure might cause autism. I don't know. Stranger things have happened. We know that fetal anticonvulsant exposure can do it but that's when mothers are actually taking the pills. It's one giant leap from that to traces in drinking water. It's the difference between falling off your chair and falling off the Empire State Building.

Michael A. Thomas, and Rebecca D. Klaper (2012). Psychoactive Pharmaceuticals Induce Fish Gene Expression Profiles Associated with Human Idiopathic Autism PLoS ONE