A Case Study in Voodoo Genetics

A new review of published studies looking at the relationship between a gene and brain structure offers a sobering lesson in how science goes wrong.

Dutch neuroscientists Marc Molendijk and colleagues took all of the studies that compared a particular variant, BDNF val66met, and the volume of the human hippocampus. It's a long story, but there are various biological reasons that these two things might be correlated.

It turns out that the first published reports found large genetic effects, but that ever since then, the size of the effects has dropped, with the latest studies finding no effects at all -

A cumulative meta-analysis confirms that as more studies on BDNF val66met have appeared, the overall effect estimate has steadily declined -

Finally, the authors found signs of publication bias: there were three small, imprecise studies that reported very large effects of the gene, but no such studies finding no effect (or a reverse effect). You'd expect that small and noisy studies would have a lot of random variation so they wouldn't all be positive even if there was a true effect; that all of the published ones were positive, suggests that null findings are out there, unreported.

Overall, this suggests that val66met probably isn't associated with hippocampus volume after all, and that the early studies showing that it was, were misleading. There's no reason to think that the early studies were wrong as such - they may have accurately reported an effect in the small sample of people they looked at, but it was only a chance finding.

I know a lot of neuroscientists who are now fairly skeptical of this whole genre of candidate gene studies; there was much excitement 5 or 10 years ago, but in retrospect, most of these studies were too small, and the publication process meant that it was the random chance findings that were most likely to get published.

However, we need to avoid any sense of complacency. Until we fix the scientific process, the same thing will happen again.

Molendijk ML, Bus BA, Spinhoven P, Kaimatzoglou A, Voshaar RC, Penninx BW, van Ijzendoorn MH, and Elzinga BM (2012). A systematic review and meta-analysis on the association between BDNF val(66) met and hippocampal volume American journal of medical genetics B Neuropsychiatric genetics PMID: 22815222