More Brain Voodoo, and This Time, It's Not Just fMRI

Ed Vul et al recently created a splash with their paper, Puzzlingly high correlations in fMRI studies of emotion, personality and social cognition (better known by its previous title, Voodoo Correlations in Social Neuroscience.) Vul et al accused a large proportion of the published studies in a certain field of neuroimaging of committing a statistical mistake. The problem, which they call the "non-independence error", may well have made the results of these experiments seem much more impressive than they should have been. Although there was no suggestion that the error was anything other than an honest mistake, the accusations still sparked a heated and ongoing debate. I did my best to explain the issue in layman's terms in a previous post.

Now, like the aftershock following an earthquake, a second paper has appeared, from a different set of authors, making essentially the same accusations. But this time, they've cast their net even more widely. Vul et al focused on only a small sub-set of experiments using fMRI to examine correlations between brain activity and personality traits. But they implied that the problem went far beyond this niche field. The new paper extends the argument to encompass papers from across much of modern neuroscience.

The article, Circular analysis in systems neuroscience: the dangers of double dipping, appears in the extremely prestigious Nature Neuroscience journal. The lead author, Dr. Nikolaus Kriegeskorte, is a postdoc in the Section on Functional Imaging Methods at the National Institutes of Health (NIH).

Kriegeskorte et al's essential point is the same as Vul et al's. They call the error in question "circular analysis" or "double-dipping", but it is the same thing as Vul et al's "non-independent analysis". As they put it, the error could occur whenever

data are first analyzed to select a subset and then the subset is reanalyzed to obtain the results.

and it will be a problem whenever the selection criteria in the first step are not independent of the reanalysis criteria in the second step. If the two sets of criteria are independent, there is no problem.


Suppose that I have some eggs. I want to know whether any of the eggs are rotten. So I put all the eggs in some water, because I know that rotten eggs float. Some of the eggs do float, so I suspect that they're rotten. But then I decide that I also want to know the average weight of my eggs . So I take a handful of eggs within easy reach - the ones that happen to be floating - and weigh them.

Obviously, I've made a mistake. I've selected the eggs that weigh the least (the rotten ones) and then weighed them. They're not representative of all my eggs. Obviously, they will be lighter than the average. Obviously. But in the case of neuroscience data analysis, the same mistake may be much less obvious. And the worst thing about the error is that it makes data look better, i.e. more worth publishing:

Distortions arising from selection tend to make results look more consistent with the selection criteria, which often reflect the hypothesis being tested. Circularity is therefore the error that beautifies results, rendering them more attractive to authors, reviewers and editors, and thus more competitive for publication. These implicit incentives may create a preference for circular practices so long as the community condones them.

To try to establish how prevalent the error is, Kriegeskorte et al reviewed all of the 134 fMRI papers published in the highly regarded journals Science, Nature, Nature Neuroscience, Neuron and the Journal of Neuroscience during 2008. Of these, they say, 42% contained at least one non-independent analysis, and another 14% may have done. That leaves 44% which were definitely "clean". Unfortunately, unlike Vul et al who did a similar review, they don't list the "good" and the "bad" papers.

They then go on to present the results of two simulated fMRI experiments in which seemingly exciting results emerge out of pure random noise, all because of the non-independence error. (One of these simulations concerns the use of pattern-classification algorithms to "read minds" from neural activity, a technique which I previously discussed). As they go on to point out, these are extreme cases - in real life situations, the error might only have a small impact. But the point, and it's an extremely important one, is that the error can creep in without being detected if you're not very careful. In both of their examples, the non-independence error is quite subtle and at first glance the methodology is fine. It's only on closer examination that the problem becomes apparent. The price of freedom from the error is eternal vigilance.

But it would be wrong to think that this is a problem with fMRI alone, or even neuroimaging alone. Any neuroscience experiment in which a large amount of data is collected and only some of it makes it into the final analysis is equally at risk. For example, many neuroscientists use electrodes to record the electrical activity in the brain. It's increasingly common to use not just one electrode but a whole array of them to record activity from more than brain one cell at once. This is a very powerful technique, but it raises the risk the non-independence error, because there is a temptation to only analyze the data from those electrodes where there is the "right signal", as the author's point out:

In single-cell recording, for example, it is common to select neurons according to some criterion (for example, visual responsiveness or selectivity) before applying
further analyses to the selected subset. If the selection is based on the same dataset as is used for selective analysis, biases will arise for any statistic not inherently independent of the selection criterion.

In fact, Kriegeskorte et al praise fMRI for being, in some ways, rather good at avoiding the problem:

To its great credit, neuroimaging has developed rigorous methods for statistical mapping from its beginning. Note that mapping the whole measurement volume avoids selection altogether; we can analyze and report results for all locations equally, while accounting for the multiple tests performed across locations..

With any luck, the publication of this paper and Vul's so close together will force the neuroscience community to seriously confront this error and related statistical weaknesses in modern neuroscience data analysis. Neuroscience can only emerge stronger from the debate.

Kriegeskorte, N., Simmons, W., Bellgowan, P., & Baker, C. (2009). Circular analysis in systems neuroscience: the dangers of double dipping Nature Neuroscience DOI: 10.1038/nn.2303

Help! There's an Epidemic of Anxiety! (Part I)

All British journalists are psychotic. Pathologically obsessed with "mental health issues", and suffering from grandiose delusions of their competence to discuss them, these demented maniacs...

Sorry. I got a bit carried away there. But you'll forgive me, because I was just following the example of seemingly everyone in the British media these past couple of weeks. If you believe the headlines, we're in the grip of an epidemic of anxiety:

BBC: UK society 'increasingly fearful'

The Telegraph: Britons 'living in fear' as record numbers suffer from anxiety

The Independent: Britain is becoming a more fearful place – and the economy is paying the price. The Indie also ran a comment by Janet Street-Porter - "The main reason people feel anxious is loneliness.", thanks Janet, qualifications: none, career path: fashion journalist - and a piece by a clinically anxious person - "I reckon a root cause of my anxiety is the modern notion that we can do away with risk by anticipating every imaginable danger."

It all started with a report by the Mental Health Foundation called In The Face Of Fear. The Mental Health Foundation are a perfectly decent charity organization, although they have a prior history of endorsing slightly dodgy research. One of their previous reports, Feeding Minds: The Impact of Food on Mental Health, presented a simplistic and overblown account of the effects of nutrition upon mood and drew heavily on the "work" of Patrick Holford, vitamin pill peddler and well-documented crank. Parts of the present report are, unfortunately, dodgy as well, as you'll see below.

In The Face of Fear is actually quite thought-provoking piece of writing, but you wouldn't know that from reading the newspapers. The headlines are all about the supposed surge in anxiety amongst the British population. This, however, is the dodgiest part of the report. Firstly, the report's authors surveyed 2246 British adults in January 2009. 37% said that they get frightened or anxious more often than they used to, 28% disagreed, and 33% neither agreed nor disagreed.

That's it. That's the finding. It's really not very impressive, because quite apart from anything else, it relies upon the respondent's ability to remember how anxious they were in the past. You just can't trust people to do hard stuff like that. I know exactly what I'm worried about today - I can't remember very well what I worried about ten years ago - so I must be more worried today! Of course, this could also work in reverse, and people might forget their past lack of anxiety and wrongly say that they are less anxious today.

The survey also found that 77% of people said that "people in general" are more anxious than they used to be, while just 3% disagreed. But remember that only (at most) 37 out of those 77% said that they themselves were actually more anxious. Hmm. So the real finding here seems to be that there is a widespread perception that other people are becoming more anxious, though it's anyone's guess whether this is in fact true. The report itself does note that

more than twice as many of us agree that people in general and the world itself are becoming more frightened and frightening as agree that they themselves are more frightened and anxious

This was rather too subtle for the newspapers, though, who reported... that people are becoming more anxious.

In The Face of Fear also cites a government study on the mental health of the British population, the Adult Psychiatric Morbidity Survey. Their use of this data, however, is selective to the point of being deception. This was a household survey of a weighted sample of the British population. That section of the population who live in houses and don't mind being interviewed about their mental health, that is. Diagnoses were made on the basis of the CIS-R interview, which scores each person on a number of symptoms (including "worry", "fatigue", and "depressive ideas"). Each person is then given a total score; a total score of 12 or more is (arbitrarily) designated to indicate a "neurotic disorder".

This was done in 1993, 2000 and 2007. The 2007 report notes that overall, levels of neurotic disorders increased between 1993 and 2000, but then stayed level in 2007. In terms of anxiety disorders, there was a very small increase in "generalized anxiety disorder" (from 4.4% to 4.7%), which mostly happened between 1993 and 2000; there was an increase in phobias, from 1993 2.2% to 2007 2.6%, but rates peaked at 2.8% in 2000; and "mixed anxiety and depressive disorder" increased from 7.5% in 1993 to 9.4% in 2000 to 9.7% in 2007.

What to make of that? It's hard to know, but it's clear that any worsening in anxiety levels occured some time between 1993 and 2000. Mysteriously, while the Mental Health Foundation report cites the 1993 and the 2007 figures, and makes much of the increase, it simply ignores and does not mention the 2000 figures, which show that any increase has long since stopped. It's history, not current events. Back in 2000, you might recall, the twin towers were still standing, The Simpsons was still funny, and Who Let The Dogs Out was top of the charts.

Overall, the evidence that people in Britian are actually feeling more and more anxious is extremely thin. In fact, I would say that it's a myth. It's a very popular myth, however: 77% of the population believe it. Why? Well, the fact that the Mental Health Foundation seem determined to make the data fit that story can't be helping matters. The newspapers, not to be outdone, focussed entirely on the scariest and most pesimissitic aspects of the report.

A poor show all round, but - as always on Neuroskeptic - there are some important lessons here about how we think about threats, social change, and "crisis". Stay tuned for the good stuff next post.

[BPSDB]

The Hollow Mask Illusion: Beyond Charlie Chaplin

Update - This paper dragon is an even better illustration of the effect, and you can make your own (if you have a printer). Amaze your friends! Really, you will.

Everyone's talking about the hollow mask illusion, a.k.a the hollow face illusion. Wired have a nice piece about this freaky visual phenomenon, complete with YouTube video so that you can see it for yourself. It's seriously weird. Here it is again, lifted from YouTube with KeepVid.com:

The illusion is a form of depth inversion. It involves a hollow (concave) object which appears to be non-hollow (convex). This happens whether the object is stationary or moving, but it's even more striking when it's in motion, as in the video above. When the mask of Charlie Chaplin rotates so that the inside is facing you, it suddenly appears as if it's looking out at you - but rotating in the opposite direction. This happens even though you know what's really going on.

The current surge of interest in the illusion was sparked by a recent fMRI study, Understanding why patients with schizophrenia do not perceive the hollow-mask illusion using dynamic causal modelling. But the fact that people with schizophrenia are generally immune to this illusion has been known for a long time. The illusion itself is even older - in fact, in one form or another, it goes back centuries.

But why exactly does it happen? That's an important question, because if you want to understand why schizophrenics are immune to it, you really need to know why it works on "normal people". (Notice that we normal people are the ones who are fooled while schizophrenics see reality as it is - R. D. Laing would be so pleased).

Most people seem to assume that the answer is pretty simple: it's expectation. We strongly expect things to be convex, so when we see something concave our brain tries to re-interpret it as convex. Easy! Hold on. There's a bit more to it than that.

The intricacies of the mask illusion are discussed in a paper called The hollow-face illusion: Object-specific knowledge, general assumptions or properties of the stimulus? The authors, Hill and Johnston, start out by discussing three possible explanations for the illusion.


First off, it could be that the illusion is driven by object-specific knowledge, i.e. knowledge about faces. (I've previously discussed the theory that faces are "special" - that our brains are specialized to percieve human faces.) According to this account, our brains expect that faces are convex, not hollow. This "top-down" expectation is so strong that it over-rides the "bottom-up" data of our eyes, and we see what we expect to see. Presumably, we also have specific expectations about teddy-bears and pineapples, which is why we see the hollow jelly moulds (above) as convex...

But Hill and Johnston point out that there's a second possibility - maybe our brains just expect everything to be convex. Rather than being about the specific object - a mask or face - the illusion might represent a more general expectation of convexity. There's some evidence for this, because there have been reports that the illusion works even for objects which the viewer has never seen before.

A variant of this explanation claims that the illusion is all about light. Maybe we expect that light always comes from above - because after all, it usually does. So we assume that the hollow mask is actually a convex face lit from above. This isn't a very good theory, however, especially because in the video above, the light actually comes from the side...

The final possible explanation considered by Hill & Johnston has nothing to do with expectation at all. Some people have claimed that the illusion occurs because the information reaching our eyes is ambiguous - it simply doesn't tell us whether the object is convex or concave. However, as Hill and Johnston point out, this is only true of information reaching one eye. We have two eyes, which gives us depth perception, meaning that we should be able to tell that the mask is hollow. Also, this wouldn't explain why the hollow mask never looks hollow. If it were ambiguous, it should be 50-50 whether it looks convex or concave.

They then go on to report the results of six different experiments investigating various aspects of the illusion. These are worth reading as they're a good example of the ways in which even something as subjective as visual illusions can be scientifically studied. After considering all of the resuls Hill & Johnston conclude

In summary, the hollow-face illusion appears to reflect a combination of the explanations offered. Some ambiguously interpretable bottom-up data must be present.That, coupled with a general bias towards convexity, is sufficient to generate the illusion, even when this interpretation is incompatible with other, unambiguous, bottom-up data. However, familiar orientations and patterns of shading and surface-colour information can greatly enhance this effect for both faces and other familiar objects.

In other words, the illusion probably is driven by expectation, but it also relies on their being some ambigious information in the first place. And while the expectations in question don't need to be about specific objects, like faces, it helps if they are.

Hill, H., & Johnston, A. (2007). The hollow-face illusion: Object-specific knowledge, general assumptions or properties of the stimulus? Perception, 36 (2), 199-223 DOI: 10.1068/p5523

Annotated Links

Sydney Spiesel writes about the myriad claimed treatments for autism in Slate. He's skeptical

If there is any illness for which 100 treatments are available, you can be sure that none of them works.

True. But he doesn't do a great job of addressing why parents swear by such ineffective treatments. His answer is the "Hawthorne Effect". I think there's rather more to it than that. For one thing, Spiesel does not consider the possibility that a treatment might have no effect at all - not even a non-specific "placebo effect" - and still become popular.

But that happens. A PLoS ONE paper, From Traditional Medicine to Witchcraft, tries to explain why. Although it features some maths and lots of graphs, the argument is summed up in a sentence

Superstitious treatments and maladaptive practices can spread because their very ineffectiveness results in sick individuals demonstrating the practice for longer than efficacious treatments, leading to more salient demonstration and more converts

In other words, the less well a treatment works, the longer it gets used, and therefore, the more likely it is for other people to see it being used and adopt it. Of course this only holds under when people are completely unable to tell whether treatments used by others work or not. This may be a valid assumption.


Psychology Today interviews rebellious British psychiatrist David Healy about his new book, Mania, which I really need to read. Healy notes that bipolar disorder became a fashionable diagnosis starting in the mid 1990s. A while back I plotted a graph showing how often bipolar disorder was mentioned in the British media. It became much more popular after about 2000 - which sort of makes sense.

Healy's one of the few people who manages to be deeply skeptical of much about modern psychiatric diagnosis and treatment while avoiding Tom Cruiseist anti-psychiatry. His last book was a homage to ECT, ferchrisakes. A lot of people felt actively betrayed by that. But if you still doubt Healy's intellect, his use in the interview of a Buffy metaphor to explain the history of "mood stabilizing drugs" should set you straight. Genius.

Depression, Neurogenesis and Herpes

Previously, I've discussed the neurogenesis theory of depression in two rather skeptical posts. Not that I'm on some kind of anti-neurogenesis theory crusade, but a study just published adds to the evidence that all's not well with that hypothesis.

The paper is Singer et. al.'s Conditional ablation and recovery of forebrain neurogenesis in the mouse. Via some cunning genetic engineering, the authors created mice with a gene for a protein called herpes simplex virus thymidine kinase. As the name suggests, this is a protein normally found in, er, herpes. Ganciclovir is a drug which can be used to treat herpes and related viral infections. And, as you might expect, cells engineered to express the herpes protein die when exposed to ganciclovir.

The authors engineered mice which expressed herpes simplex virus thymidine kinase, but only in neural progenitor cells. These are the cells which eventually become new neurones in the adult brain. They found that injections of gancyclovir devasasted the production of new neurones in the engineered mice. (It had no effect on normal mice, of course, because their brain cells weren't half mouse, half herpes). That's not all that surprising.

However, they also found that gancyclovir treatment had no effect on the ability of 28 days treatment imipramine, an antidepressant, to affect the mice's behaviour. (The measure of antidepressant action was the Tail Suspension Test). That's a result, because a lot of people are interested in the theory that antidepressants work by boosting neurogenesis in the hippocampus. If that were true, blocking neurogenesis should also block the effects of antidepressants.

Some rather exciting experiments found that it does, most famously the much-cited Santarelli et al (2003). But a growing number of other studies, such as this one, have not confirmed this finding. This doesn't mean that Santarelli et al were wrong, but it does suggest that there's more to antidepressants than neurogenesis. The seemingly-contradictory findings of the various studies might be due to important differences in the methods used. For example, the authors of this paper say that Santarelli et al's way of blocking neurogenesis - using x-rays - may have also caused inflammation and blocked the formation of non-neural cells, such as those which go to make up blood-vessels.

Of course, it's easy enough for us to speculate along such lines - rather harder to work out what exactly is going on. With any luck, the next few years will see more progress on this important topic.

Singer, B., Jutkiewicz, E., Fuller, C., Lichtenwalner, R., Zhang, H., Velander, A., Li, X., Gnegy, M., Burant, C., & Parent, J. (2009). Conditional ablation and recovery of forebrain neurogenesis in the mouse The Journal of Comparative Neurology, 514 (6), 567-582 DOI: 10.1002/cne.22052

More on that Homeopathy Analysis

As promised, I emailed the authors of that Cochrane Review of homeopathy for reducing the side effects of cancer treatment. I asked them to clarify why they had included the Pommier et al trial of Calendula ointment - a decision which attracted some criticism. Their (very prompt) response included this statement:

"...We contacted the manufacturer of the calendula ointment and they confirmed that it had been prepared in accordance with the German Homeopathic Pharmacopoeia, therefore the trial met our inclusion criteria..."

Which is a reference to the inclusion criteria as set out in their paper, i.e.:

Homeopathy (also spelt homoeopathy) was defined, for the purpose of this review, as the use of homeopathic medicines prepared in accordance with officially recognised homeopathic pharmacopoeias. Where there was doubt about the classification of the medicine, we contacted authors or the product manufactures for confirmation. Any homeopathic prescribing strategy was included (pp. 3-4).

So it looks like the debate over the inclusion of this trial boils down to a difference of opinion over the definition of "homeopathy". Critics (myself included) who questioned the inclusion of this trial did so because we hold a rather narrower concept of "homeopathy" than the authors do. Of course, there is no right or wrong definition of homeopathy - no-one holds a trademark on the term - so I think that this is where the debate is going to have to rest.

[BPSDB]

BBC: Homeopathy Works, Oh Wait...

According to a BBC headline today, the Cochrane Collaboration says that:

Homeopathy 'eases cancer therapy'

Wow. The Cochrane Collaboration, the very embodiment of evidence-based medicine, says that homeopathy works! That would really be something to write home about. If it were true. What a recent Cochrane Review, Homeopathic medicines for adverse effects of cancer treatments, in fact concluded was that:

This review found preliminary data in support of the efficacy of topical calendula for prophylaxis of acute dermatitis during radiotherapy and Traumeel S mouthwash in the treatment of chemotherapy-induced stomatitis. These trials need replicating. There is no convincing evidence for the efficacy of homeopathic medicines for other adverse effects of cancer treatments. Further research is required.

In other words, they found two high-quality positive trials. One of them, the trial of "Traumeel S", included just 30 patients, 15 on placebo and 15 on the homeopathic treatment. Which is not very many. Still, at least it actually was a trial of a homeopathic treatment. The other positive trial, the one on "topical calendula", wasn't.

No, really. The second trial, a fairly large French study (254 patients), used an ointment made from a herb, Calendula officinalis, aka the Pot Marigold. Unlike homeopathic treatments, which are just water, the Calendula ointment used in the study in question was apparantly

fabricated from a plant of the marigold family, Calendula officinalis. The digest is obtained by incubation at 75°C in petroleum jelly to extract the liposoluble components of the plant.

In other words, it contained plenty of chemicals from the plant, and would be better described as a herbal product, not a homeopathic one. Unlike in a homeopathic remedy, the herb wasn't diluted in water several times until no molecules of the original product remained. It's not homeopathy. The word "homeopathy" doesn't appear anywhere in the paper!

So why on earth was this study included in the Cochrane Review? This is where things get weird. The Cochrane authors describe the paper as "a study of a homeopathic ointment". But how did they even hear about the paper, given that the word homeopathy appears nowhere in the paper, or the abstract, or the PubMed keywords? They say that "One potential study was identified by an expert in the field (Pommier 2004)."

So the best evidence that homeopathy works for alleviating the symptoms of cancer therapy is a paper that isn't about homeopathy, identified by "an expert" in homeopathy. And the Cochrane Collaboration took his word for it. And the BBC reported on it as fact (although to their credit they do include a scathing comment from Prof. Edward Ernst.) A great job all round!

If anyone has any ideas about why this paper was included in the Cochrane review, I'd be interested to hear them. I will be in contact with the authors of the review to try to find out why, because if the Cochrane Collaboration has really just published a review about homeopathy which includes a trial which is nothing to do with homeopathy, it's something of a scandal. I find it somewhat hard to believe just on the basis that Cochrane reviews are generally very competent. So, stay tuned for more on this.

Update 18 4 2009: The authors have kindly responded to my email.

[BPSDB]

The Voodoo Strikes Back

Just when you thought it was safe to compute a correlatation between a behavioural measure and a cluster mean BOLD change...

The fMRI voodoo correlations controversy isn't over. Ed Vul and collegues have just responded to their critics in a new article (pdf). The critics appear to have scored at least one victory, however, since the original paper has now been renamed. So it's goodbye to "Voodoo Correlations in Social Neuroscience" - now it's "Puzzlingly high correlations in fMRI studies of emotion, personality and social cognition" by Vul et. al. 2009. Not quite as catchy, but then, that's the point...

Just in case you need reminding of the story so far: A couple of months ago, MIT grad student Ed Vul and co-authors released a pre-publication manuscript, then titled Voodoo Correlations in Social Neuroscience. This paper reviewed the findings of a number of fMRI studies which reported linear correlations between regional brain activity and some kind of measure of personality. Vul et. al. argued that many (but by no means all) of these correlations were in fact erroneous, with the reported correlations being much higher than the true ones. Vul et. al. alleged that the problem arose due to a flaw in the statistical analysis used, the "non-independence error". For my non-technical explanation of the issue, see my previous post, or go read the original paper (it really doesn't require much knowledge of statistics).

Vul's paper attracted a lot of praise and also a lot of criticism, both in the blogosphere and in the academic literature. Many complained that it was sensationalistic and anti-fMRI. Others embraced it for the same reasons. My view was that while the paper's style was certainly journalistic, and while many of those who praised the paper did so for the wrong reasons, the core argument was both valid and important. While not representing a radical challenge to social neuroscience or fMRI in general, Vul et. al. draws attention to a widespread and potentially serious technical issue with the analysis of fMRI data, one which all neuroscientists should be aware of.

That's still my opinion. Vul et. al.'s response to their critics is a clearly worded and convincing defense. Interestingly, their defense is in many ways just a clarificiation of the argument. This is appropriate, because I think the argument is pretty much just common sense once it is correctly understood. As far as I can see the only valid defence against it is to say that a particular paper did not in fact commit the error - while not disputing that the error itself is a problem. Vul et. al. say that to their knowledge no accused papers have turned out to be innocent - although I'm sure we haven't heard the last of that.

Vul et. al. also now make explicit something which wasn't very clear in their original paper, namely that the original paper made accusations of two completely seperate errors. One, the non-independence error, is common but probably less serious than the second, the "Forman error", which is pretty much fatal. Fortunately, so far, only two papers are known to have fallen prey to the Forman error - although there could be more. Go read the article for more details on what could be Vul's next bombshell...

EDWARD VUL, CHRISTINE HARRIS, PIOTR WINKIELMAN, AND, & HAROLD PASHLER (2009). Reply to comments on “Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition” Perspectives in Psychological Science

Antidepressants: Clinical Trials versus Real Life

In a recent post, I argued that no-one knows how well antidepressants work. Although there have been a huge number of clinical trials conducted on a variety of antidepressant drugs, it is impossible to know what the results of these trials mean in terms of real benefits for real patients.

I'm not the only skeptic. A paper just out in the American Journal of Psychiatry adds to growing case against contemporary antidepressant trials (almost all of which are industry-sponsored) and should give everyone cause for thought.
The article, Can Phase III Trial Results of Antidepressant Medications Be Generalized to Clinical Practice? A STAR*D Report, is one of the many spin-offs from STAR*D. STAR*D was a large and ambitious study designed to investigate the effectiveness of antidepressants in a realistic setting. The results were rather difficult to interpret (and some are yet to be published), but this report is certainly amongst the most interesting.

One of the things that made STAR*D different from the average trial was the recruitment criteria. Most trials require a volunteer to tick numerous boxes before they can be enrolled - for example, it's common to exclude anyone showing signs of suicidal thoughts or behaviours, people with any problems other than depression, such as addictions, and anyone whose depression is rated as "insufficiently severe" using a depression rating scale such as the HAMD.

The majority, probably the vast majority, of people who suffer from depression don't fit such narrow criteria. So antidepressants end up being tested on no more than a small select group of the people who are likely to end up taking them when and if they hit the market. And because criteria can differ between trials, two trials might end up testing the same drug on two quite different types of people - although on paper they are both a trial of the drug for the exact same thing, "major depression".

To be fair, some criteria are necessary to protect the safety of volunteers (you don't want someone who is suicidal getting their hands on an experimental and potentially dangerous drug), but the whole situation is far from ideal. People have been complaining about it for a while. The new paper adds to the list of complaints. The authors took advantage of the fact that STAR*D did not have restrictive entry criteria, and simply compared those patients who did happen to fit the bill for a "typical" antidepressant trial vs. those who didn't.

First off, just under a quarter (22.2%) of patients met the typical criteria. That's really not very many. And, as you'd expect, this minority of patients were rather different from the rest. Amongst many other things they were slightly younger, a lot richer (mean monthly income $3050 vs. $2163), much less likely to be unemployed or to have no medical insurance, and less likely to be black or Hispanic (this was an American sample).

Such differences might seem unimportant - if someone is suffering from a disease, and they're given a medication to treat it, does the size of their paycheque really matter? Yes it could - the patients who met the criteria for a typical antidepressant trial reported on average more improvement, and fewer side effects, compared to the others. (They were all given citalopram, a popular and pretty decent SSRI).

Does this mean that rich white people really get more benefit from citalopram? Or do they just tend to report more benefit? Or do they experience larger placebo effects? It's impossible to say. The authors, who include some big names in antidepressant research, conclude that:

...a patient sample that meets the inclusion criteria for a phase III clinical trial is not representative of depressed patients seen in typical clinical practice, and phase III trial outcomes may be more optimistic than results obtained in practice.

Although it's also possible that trial outcomes could be more pessimistic, in terms of finding smaller drug-placebo differences than they otherwise would. Only one thing is certain - antidepressant trials are far removed from the real world, and the results of such trials have to be taken with a large pinch of salt.

Wisniewski, S., Rush, A., Nierenberg, A., Gaynes, B., Warden, D., Luther, J., McGrath, P., Lavori, P., Thase, M., Fava, M., & Trivedi, M. (2009). Can Phase III Trial Results of Antidepressant Medications Be Generalized to Clinical Practice? A STAR*D Report American Journal of Psychiatry DOI: 10.1176/appi.ajp.2008.08071027

A Couple of Links

A couple of neat things I discovered this week:

Just judging by the name, you might think that ScienceWatch was one of those tedious attack sites going under the guise of "watches" (naming no names). But it's actually about "tracking trends and performance in basic research". By analysing citation data and so forth, they claim to be able to identify "hot papers" and, more interesting, hot "fronts" or themes in research. It's a commercial enterprise, but a lot of the material is free. They just mapped out the hot topics in current OCD research (although the results were hardly surprising).

Then there's Pology Magazine, which is a travel magazine, but with an anthropological/social science approach - and lots of extremely pretty pictures. It's well worth a visit.