In Science, Popularity Means Inaccuracy

Who's more likely to start digging prematurely: one guy with a metal-detector looking for an old nail, or a field full of people with metal-detectors searching for buried treasure?

In any area of science, there will be some things which are more popular than others - maybe a certain gene, a protein, or a part of the brain. It's only natural and proper that some things get of lot of attention if they seem to be scientifically important. But Thomas Pfeiffer and Robert Hoffmann warn in a PLoS One paper that popularity can lead to inaccuracy - Large-Scale Assessment of the Effect of Popularity on the Reliability of Research.

They note two reasons for this. Firstly, popular topics tend to attract interest and money. This means that scientists have much to gain by publishing "positive results" as this allows them to get in on the action -

In highly competitive fields there might be stronger incentives to “manufacture” positive results by, for example, modifying data or statistical tests until formal statistical significance is obtained. This leads to inflated error rates for individual findings... We refer to this mechanism as “inflated error effect”.

Secondly, in fields where there is a lot of research being done, the chance that someone will, just by chance, come up with a positive finding increases -

The second effect results from multiple independent testing of the same hypotheses by competing research groups. The more often a hypothesis is tested, the more likely a positive result is obtained and published even if the hypothesis is false. ... We refer to this mechanism as “multiple testing effect”.

But does this happen in real life? The authors say yes, based on a review of research into protein-protein interactions in yeast. (Happily, you don't need to be a yeast expert to follow the argument.)

There are two ways of trying to find out whether two proteins interact with each other inside cells. You could do a small-scale experiment specifically looking for one particular interaction: say, Protein B with Protein X. Or you can do "high-throughput" screening of lots of proteins to see which ones interact: Does Protein A interact with B, C, D, E... Does Protein B interact with A, C, D, E... etc.

There have been tens of thousands of small-scale experiments into yeast proteins, and more recently, a few high-throughput studies. The authors looked at the small-scale studies and found that the more popular a certain protein was, the less likely it was that reported interactions involving it would be confirmed by high-throughput experiments.

The second and the third of the above graphs shows the effect. Increasing popularity leads to a falling % of confirmed results. The first graph shows that interactions which were replicated by lots of small-scale experiments tended to be confirmed, which is what you'd expect.

Pfeiffer and Hoffmann note that high-throughput studies have issues of their own, so using them as a yardstick to judge the truth of other results is a little problematic. However, they say that the overall trend remains valid.

This is an interesting paper which provides some welcome empirical support to the theoretical argument that popularity could lead to unreliability. Unfortunately, the problem is by no means confined to yeast. Any area of science in which researchers engage in a search for publishable "positive results" is vulnerable to the dangers of publication bias, data cherry-picking, and so forth. Even obscure topics are vulnerable but when researchers are falling over themselves to jump on the latest scientific bandwagon, the problems multiply exponentially.

A recent example may be the "depression gene", 5HTTLPR. Since a landmark paper in 2003 linked it to clinical depression, there has been an explosion of research into this genetic variant. Literally hundreds of papers appeared - it is by far the most studied gene in psychiatric genetics. But a lot of this research came from scientists with little experience or interest in genes. It's easy and cheap to collect a DNA sample and genotype it. People started routinely looking at 5HTTLPR whenever they did any research on depression - or anything related.

But wait - a recent meta-analysis reported that the gene is not in fact linked to depression at all. If that's true (it could well be), how did so many hundreds of papers appear which did find an effect? Pfeiffer and Hoffmann's paper provides a convincing explanation.

Link - Orac also blogged this paper and put a characteristic CAM angle on it.

Pfeiffer, T., & Hoffmann, R. (2009). Large-Scale Assessment of the Effect of Popularity on the Reliability of Research PLoS ONE, 4 (6) DOI: 10.1371/journal.pone.0005996

Everyone is Mentally Ill

There's been a lot of interest over the idea that an "Artificial brain is 10 years away", which is what Professor Henry Markram told the ultra-hip TED conference in Oxford the other day.

That's an amazing idea. But Markram said something else even more astonishing, which, for some reason, has not got nearly as much attention:

"There are two billion people on the planet affected by mental disorder," he told the audience.

Two billion people. One in three.

This was presumably a throw-away remark, something he said in order to emphasise the importance of understanding the brain. But this makes it even more amazing: we have reached the point where no-one bats an eyelid at the idea that mental illness affects one in three people worldwide.

Well, if this is what we believe now, I think we need to stop beating about the bush with numbers like one in four or one in three, and admit that we now are now using "mental illness" as a synonym for "the human condition".

After all, once you pass the point where one in two people have something, you are saying that it's normal and not having it is weird. As I've written before, if you take the evidence seriously, more than 50% of people are indeed "mentally" ill at some point. So let's just say that everyone is mentally ill and have done with it.

Or we could reassess what we mean by "mental illness" and stop medicalizing human suffering. Hey, we can dream.

Major Huntington's Disease Puzzle Solved

Huntington's Disease is a genetic neurological disorder. Symptoms most commonly appear around age 40, and they progress gradually from subtle movement abnormalities to complete loss of motor control and dementia. Psychiatric problems, especially depression and irritability, are also common and may be the first signs. Treatment consists of medications to mask some of the symptoms. Singer Woodie Guthrie is perhaps the disease's best known victim: he ended his days in a mental institution.

Huntington's results from mutations in the gene which makes a protein called Huntingtin (Htt). The symptoms are associated with degeneration of various parts of the brain, most importantly, a deep region called the striatum. These facts have been known for many years, but it's unclear how mutant Htt damages the brain. There are various theories, but they've all faced a serious puzzle - Htt is expressed in all adult human cells, but Huntington's only affects some neurones. Why?

A major new paper claims to have the answer : Rhes, a Striatal Specific Protein, Mediates Mutant-Huntingtin Cytotoxicity. It also suggests a promising target for drugs that could prevent the damage from occuring.

The authors assemble evidence showing that mutant Htt kills cells only in conjunction with another protein called Rhes. Crucially, Rhes is only expressed in striatal cells. They found that:

• Rhes binds to Htt, but it binds much more strongly to mutant Htt.
• Causing cells to express both Rhes and mutant Htt leads to cell death, but either by its own does not.
• Rhes mediates the binding of mutant Htt to another protein, SUMO, which causes the mutant Htt to become more soluble and therefore more toxic to cells.

This looks to be a very important addition to the literature on Huntington's. The implication is that a drug which could prevent Rhes from SUMOylating mutant Htt would halt the progression of the disease (although it would presumably not reverse any damage which had already happened.) This is the kind of powerful explanation that neuroscientists who study psychiatric disorders dream about. Maybe in 50 years we will have a similar understanding of schizophrenia - maybe.

Huntington's is a fascinating disorder. The mechanism of inheritance is very distinctive - disease results when a certain section of DNA is too long, and the longer it is, the earlier and more severe the symptoms. And when the pathogenic region is too long, it tends to get even longer during the formation of sperm cells, so the children of fathers with Huntington's often suffer from a more severe, early-onset form. This phenemonon is called genetic anticipation and is unique to Huntingdon's and some similar disorders.

Huntington's is also one of the few disorders which can be accurately diagnosed genetically before the symptoms occur. Anyone at risk of the disease can take a DNA test and know their fate. Perhaps unsurprisingly, most choose not to.

Subramaniam, S., Sixt, K., Barrow, R., & Snyder, S. (2009). Rhes, a Striatal Specific Protein, Mediates Mutant-Huntingtin Cytotoxicity Science, 324 (5932), 1327-1330 DOI: 10.1126/science.1172871

The Onion Does China

The Onion turns its satirical eye on China, with hilarious if not entirely PC results -
Here's a screenshot for posterity, because their "special issues" tend to go back to normal pretty quickly.

I always think it's a little odd that the Chinese government don't have anyone whose default assumption is that they're in the right. Whenever a Western or a Western-aligned country does something morally... questionable, you can count on conservatives to defend it. Whereas countries with a history of Western exploitation generally enjoy the benefit of the liberal doubt. But China, almost uniquely, gets it from left, right, and centre equally.

I remember a colleague's astonishment when a Chinese post-doc expressed the opinion that Tibet was part of China and should remain so. This was an idea that she'd just never heard before, and she clearly thought it was entirely bizarre. Yet it was only 40 years ago that many French people were of the opinion that L’Algérie est française et le restera - Algeria! And there are still people in Northern Ireland who might kill you if you suggest that that province doesn't belong to Britain.

More on Suppressed Clinical Trials

We read in the BMJ that a German agency refuses to rule on drug’s benefits until Pfizer discloses all trial results. The drug is reboxetine (Edronax), which readers will recall was recently deemed to be the worst new antidepressant by an Oxford team.


The agency, the IQWiG, are an independent organization, but they were comissioned by the German federal government to report on the benefits of three antidepressants: reboxetine, mirtazapine, and buproprion. Their decision will have major implications in terms of which drugs are available through the German public healthcare system. However, they decided to suspend judgement on reboxetine because the manufacturers, Pfizer, did not provide them with the results of all available trials.

Ten relevant trials for reboxetine that could definitely be included were identified from the literature search in bibliographic databases, publicly accessible drug approval documents, and clinical trial registries. However, 3 of these trials could not be analysed with regard to the antidepressive effect of reboxetine because the publications only contained data on partial populations ...

Furthermore, 6 potentially relevant trials were identified that could not be included because no full publication existed and the manufacturer of reboxetine (Pfizer) refused to provide full information on all trials with reboxetine...

[and] Due to insufficient cooperation from the manufacturer of reboxetine, it remained unclear whether additional unpublished trials exist. It may well be that the identified data represent an even smaller portion of available evidence.

IQWiG have pulled no punches. It's not every day that a healthcare agency puts out a press release headlined "Pfizer conceals study data: Drug manufacturer hinders the best possible treatment of patients with depression". And to be fair to Pfizer, they are hardly behaving worse than their rivals. Essex pharmaceuticals, the manufacturers of mirtazapine, also failed to provide all the data on their drug. GlaxoSmithKline, who make buproprion, did in fact play ball in this case, but this is the very same company who have been widely accused of suppressing data on Seroxat-induced suicides in kids.

So the problem is not unique to Pfizer, or to reboxetine. The problem is with the system which allows drug manufacturers to conduct as many trials as they like and only publish the results they want. The resulting publication bias is damaging to every field of medicine. One solution is for scientific journals to only publish trials that were pre-registered before they started, so that everyone knows about trials before they happen. But maybe this isn't enough. The IQWiG note that the US has a law mandating that clinical trial results must be registered and reported to the FDA, and call for the EU to adopt similar legislation.

Stafford, N. (2009). German agency refuses to rule on drug's benefits until Pfizer discloses all trial results BMJ, 338 (jun22 1) DOI: 10.1136/bmj.b2521