More on that Homeopathy Analysis

As promised, I emailed the authors of that Cochrane Review of homeopathy for reducing the side effects of cancer treatment. I asked them to clarify why they had included the Pommier et al trial of Calendula ointment - a decision which attracted some criticism. Their (very prompt) response included this statement:

"...We contacted the manufacturer of the calendula ointment and they confirmed that it had been prepared in accordance with the German Homeopathic Pharmacopoeia, therefore the trial met our inclusion criteria..."

Which is a reference to the inclusion criteria as set out in their paper, i.e.:

Homeopathy (also spelt homoeopathy) was defined, for the purpose of this review, as the use of homeopathic medicines prepared in accordance with officially recognised homeopathic pharmacopoeias. Where there was doubt about the classification of the medicine, we contacted authors or the product manufactures for confirmation. Any homeopathic prescribing strategy was included (pp. 3-4).

So it looks like the debate over the inclusion of this trial boils down to a difference of opinion over the definition of "homeopathy". Critics (myself included) who questioned the inclusion of this trial did so because we hold a rather narrower concept of "homeopathy" than the authors do. Of course, there is no right or wrong definition of homeopathy - no-one holds a trademark on the term - so I think that this is where the debate is going to have to rest.

[BPSDB]

BBC: Homeopathy Works, Oh Wait...

According to a BBC headline today, the Cochrane Collaboration says that:

Homeopathy 'eases cancer therapy'

Wow. The Cochrane Collaboration, the very embodiment of evidence-based medicine, says that homeopathy works! That would really be something to write home about. If it were true. What a recent Cochrane Review, Homeopathic medicines for adverse effects of cancer treatments, in fact concluded was that:

This review found preliminary data in support of the efficacy of topical calendula for prophylaxis of acute dermatitis during radiotherapy and Traumeel S mouthwash in the treatment of chemotherapy-induced stomatitis. These trials need replicating. There is no convincing evidence for the efficacy of homeopathic medicines for other adverse effects of cancer treatments. Further research is required.

In other words, they found two high-quality positive trials. One of them, the trial of "Traumeel S", included just 30 patients, 15 on placebo and 15 on the homeopathic treatment. Which is not very many. Still, at least it actually was a trial of a homeopathic treatment. The other positive trial, the one on "topical calendula", wasn't.

No, really. The second trial, a fairly large French study (254 patients), used an ointment made from a herb, Calendula officinalis, aka the Pot Marigold. Unlike homeopathic treatments, which are just water, the Calendula ointment used in the study in question was apparantly

fabricated from a plant of the marigold family, Calendula officinalis. The digest is obtained by incubation at 75°C in petroleum jelly to extract the liposoluble components of the plant.

In other words, it contained plenty of chemicals from the plant, and would be better described as a herbal product, not a homeopathic one. Unlike in a homeopathic remedy, the herb wasn't diluted in water several times until no molecules of the original product remained. It's not homeopathy. The word "homeopathy" doesn't appear anywhere in the paper!

So why on earth was this study included in the Cochrane Review? This is where things get weird. The Cochrane authors describe the paper as "a study of a homeopathic ointment". But how did they even hear about the paper, given that the word homeopathy appears nowhere in the paper, or the abstract, or the PubMed keywords? They say that "One potential study was identified by an expert in the field (Pommier 2004)."

So the best evidence that homeopathy works for alleviating the symptoms of cancer therapy is a paper that isn't about homeopathy, identified by "an expert" in homeopathy. And the Cochrane Collaboration took his word for it. And the BBC reported on it as fact (although to their credit they do include a scathing comment from Prof. Edward Ernst.) A great job all round!

If anyone has any ideas about why this paper was included in the Cochrane review, I'd be interested to hear them. I will be in contact with the authors of the review to try to find out why, because if the Cochrane Collaboration has really just published a review about homeopathy which includes a trial which is nothing to do with homeopathy, it's something of a scandal. I find it somewhat hard to believe just on the basis that Cochrane reviews are generally very competent. So, stay tuned for more on this.

Update 18 4 2009: The authors have kindly responded to my email.

[BPSDB]

The Voodoo Strikes Back

Just when you thought it was safe to compute a correlatation between a behavioural measure and a cluster mean BOLD change...

The fMRI voodoo correlations controversy isn't over. Ed Vul and collegues have just responded to their critics in a new article (pdf). The critics appear to have scored at least one victory, however, since the original paper has now been renamed. So it's goodbye to "Voodoo Correlations in Social Neuroscience" - now it's "Puzzlingly high correlations in fMRI studies of emotion, personality and social cognition" by Vul et. al. 2009. Not quite as catchy, but then, that's the point...

Just in case you need reminding of the story so far: A couple of months ago, MIT grad student Ed Vul and co-authors released a pre-publication manuscript, then titled Voodoo Correlations in Social Neuroscience. This paper reviewed the findings of a number of fMRI studies which reported linear correlations between regional brain activity and some kind of measure of personality. Vul et. al. argued that many (but by no means all) of these correlations were in fact erroneous, with the reported correlations being much higher than the true ones. Vul et. al. alleged that the problem arose due to a flaw in the statistical analysis used, the "non-independence error". For my non-technical explanation of the issue, see my previous post, or go read the original paper (it really doesn't require much knowledge of statistics).

Vul's paper attracted a lot of praise and also a lot of criticism, both in the blogosphere and in the academic literature. Many complained that it was sensationalistic and anti-fMRI. Others embraced it for the same reasons. My view was that while the paper's style was certainly journalistic, and while many of those who praised the paper did so for the wrong reasons, the core argument was both valid and important. While not representing a radical challenge to social neuroscience or fMRI in general, Vul et. al. draws attention to a widespread and potentially serious technical issue with the analysis of fMRI data, one which all neuroscientists should be aware of.

That's still my opinion. Vul et. al.'s response to their critics is a clearly worded and convincing defense. Interestingly, their defense is in many ways just a clarificiation of the argument. This is appropriate, because I think the argument is pretty much just common sense once it is correctly understood. As far as I can see the only valid defence against it is to say that a particular paper did not in fact commit the error - while not disputing that the error itself is a problem. Vul et. al. say that to their knowledge no accused papers have turned out to be innocent - although I'm sure we haven't heard the last of that.

Vul et. al. also now make explicit something which wasn't very clear in their original paper, namely that the original paper made accusations of two completely seperate errors. One, the non-independence error, is common but probably less serious than the second, the "Forman error", which is pretty much fatal. Fortunately, so far, only two papers are known to have fallen prey to the Forman error - although there could be more. Go read the article for more details on what could be Vul's next bombshell...

EDWARD VUL, CHRISTINE HARRIS, PIOTR WINKIELMAN, AND, & HAROLD PASHLER (2009). Reply to comments on “Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition” Perspectives in Psychological Science

Antidepressants: Clinical Trials versus Real Life

In a recent post, I argued that no-one knows how well antidepressants work. Although there have been a huge number of clinical trials conducted on a variety of antidepressant drugs, it is impossible to know what the results of these trials mean in terms of real benefits for real patients.

I'm not the only skeptic. A paper just out in the American Journal of Psychiatry adds to growing case against contemporary antidepressant trials (almost all of which are industry-sponsored) and should give everyone cause for thought.
The article, Can Phase III Trial Results of Antidepressant Medications Be Generalized to Clinical Practice? A STAR*D Report, is one of the many spin-offs from STAR*D. STAR*D was a large and ambitious study designed to investigate the effectiveness of antidepressants in a realistic setting. The results were rather difficult to interpret (and some are yet to be published), but this report is certainly amongst the most interesting.

One of the things that made STAR*D different from the average trial was the recruitment criteria. Most trials require a volunteer to tick numerous boxes before they can be enrolled - for example, it's common to exclude anyone showing signs of suicidal thoughts or behaviours, people with any problems other than depression, such as addictions, and anyone whose depression is rated as "insufficiently severe" using a depression rating scale such as the HAMD.

The majority, probably the vast majority, of people who suffer from depression don't fit such narrow criteria. So antidepressants end up being tested on no more than a small select group of the people who are likely to end up taking them when and if they hit the market. And because criteria can differ between trials, two trials might end up testing the same drug on two quite different types of people - although on paper they are both a trial of the drug for the exact same thing, "major depression".

To be fair, some criteria are necessary to protect the safety of volunteers (you don't want someone who is suicidal getting their hands on an experimental and potentially dangerous drug), but the whole situation is far from ideal. People have been complaining about it for a while. The new paper adds to the list of complaints. The authors took advantage of the fact that STAR*D did not have restrictive entry criteria, and simply compared those patients who did happen to fit the bill for a "typical" antidepressant trial vs. those who didn't.

First off, just under a quarter (22.2%) of patients met the typical criteria. That's really not very many. And, as you'd expect, this minority of patients were rather different from the rest. Amongst many other things they were slightly younger, a lot richer (mean monthly income $3050 vs. $2163), much less likely to be unemployed or to have no medical insurance, and less likely to be black or Hispanic (this was an American sample).

Such differences might seem unimportant - if someone is suffering from a disease, and they're given a medication to treat it, does the size of their paycheque really matter? Yes it could - the patients who met the criteria for a typical antidepressant trial reported on average more improvement, and fewer side effects, compared to the others. (They were all given citalopram, a popular and pretty decent SSRI).

Does this mean that rich white people really get more benefit from citalopram? Or do they just tend to report more benefit? Or do they experience larger placebo effects? It's impossible to say. The authors, who include some big names in antidepressant research, conclude that:

...a patient sample that meets the inclusion criteria for a phase III clinical trial is not representative of depressed patients seen in typical clinical practice, and phase III trial outcomes may be more optimistic than results obtained in practice.

Although it's also possible that trial outcomes could be more pessimistic, in terms of finding smaller drug-placebo differences than they otherwise would. Only one thing is certain - antidepressant trials are far removed from the real world, and the results of such trials have to be taken with a large pinch of salt.

Wisniewski, S., Rush, A., Nierenberg, A., Gaynes, B., Warden, D., Luther, J., McGrath, P., Lavori, P., Thase, M., Fava, M., & Trivedi, M. (2009). Can Phase III Trial Results of Antidepressant Medications Be Generalized to Clinical Practice? A STAR*D Report American Journal of Psychiatry DOI: 10.1176/appi.ajp.2008.08071027

A Couple of Links

A couple of neat things I discovered this week:

Just judging by the name, you might think that ScienceWatch was one of those tedious attack sites going under the guise of "watches" (naming no names). But it's actually about "tracking trends and performance in basic research". By analysing citation data and so forth, they claim to be able to identify "hot papers" and, more interesting, hot "fronts" or themes in research. It's a commercial enterprise, but a lot of the material is free. They just mapped out the hot topics in current OCD research (although the results were hardly surprising).

Then there's Pology Magazine, which is a travel magazine, but with an anthropological/social science approach - and lots of extremely pretty pictures. It's well worth a visit.