How Not To Do Great Science (The Lost Post)

This post was originally published on Discover Magazine on September 16th 2013, but has since vanished (although most of my other Discover posts are still available). Luckily, I saved a backup. So here's the original "How Not To Do Great Science".

---

This post is a bit special. For the first time ever, I've collaborated with an artist, Erene Stergiopoulos. Her webcomic is here and she's on Twitter here. I think you'll agree that the artistic standard is a little higher than I usually achieve. Anyway, here's what we did:

It would be silly to expect that every architect should finish buildings at a certain rate. That would make it impossible to anyone to build certain things. Some things take longer to build than others, and most great things take a great deal of time. Faced with a sufficiently demanding quota, builders might be reduced to rushing out follies that might look impressive from a distance, but that are no more than hollow shells. Yet, as silly it would be to make uniform demands of architects, this is what is happening to scientists.

Rather than build, scientists are expected to publish - and publish fast - or perish. My worry (and that of many others) is that the pressure to publish often fundamentally changes not just how much scientists write, but what they can write about. It turns researchers into prolific doers of small deeds, but it leaves them little time to think about, let alone complete, great works. Though the mills of God grind slowly...

Yet the problem is not just the speed of science today, but also the direction: go to a scientific conference and you'll see perfectly good data in the process of being oversold, misinterpreted, and p-hacked into a 'publishable' form.

Much has been said about how this leads to false positives - impressive follies that don't stand up to scrutiny. What's less discussed - and the point of this piece - is the opportunity cost. New theories come out of attempts to explain 'negative' data - negative from the perspective of the old theory. Null results are the foundations of future progress, but only if they are allowed to lie there awhile; not if they are torn up and used to prop up tottering old structures.

Neuroskeptic Has Moved

After four years, Neuroskeptic is no longer an indie blog - I've moved to Discover Magazine blogs.

The new address is here: http://blogs.discovermagazine.com/neuroskeptic/

So please stop using this blogspot site; don't link to it, don't visit it, etc. All the posts are on the new one. The new RSS feed is: http://feeds.feedburner.com/neuro-skeptic/
Comments on this blog are now turned off but all old comments have been transferred to the new site, and you can comment there.

Neuroskeptic Is Moving

Over the next few days, Neuroskeptic will be moving to Discover Magazine. Which is very exciting.

This will mean no new posts for at least a week, while we sort out the technical issues of the transition, and I've also turned comments off - all existing comments will be moved over to the new blog, however.

For more updates you can follow me on Twitter...

See you on the other side soon!

Still 'Profiteering From Anxiety'

Late last year, the excellent Neurobonkers blog covered a case of 'Profiteering from anxiety'.

It seems one Nader Amir has applied for a patent on the psychological technique of 'Attentional Retraining', a method designed to treat anxiety and other emotional problems by conditioning the mind to unconsciously pay more attention to positive things and ignore unpleasant stuff.

For just $139.99, you can have a crack at modifying your unconscious with the help of Amir's Cognitive Retraining Technologies.

It's a clever idea... but hardly a new one. As Neurobonkers said, research on these kinds of methods had been going on for years before Amir came on the scene. In a comment, Prof. Colin MacLeod (who's been researching this stuff for over 20 years) argued that "I do not believe that a US patent granted to Prof Amir for the attentional bias modification approach would withstand challenge."

Well, in an interesting turn of events, Amir has issued just Corrections (1,2) to two of his papers. Both of the articles reported that retraining was an effective treatment for anxiety; but in both cases he now reveals that there was

an error...in the article a disclosure should have been noted that Nader Amir is the co-founder of a company that markets anxiety relief products.

Omitting to declare a conflict of interest... how unfortunate.

Still, it's an easy mistake to make: when you're focused on doing unbiased, objective, original research, as Amir doubtless was, such mundane matters are the last thing you tend to pay attention to.

Amir, N., and Taylor, C. (2013). Correction to Amir and Taylor (2012). Journal of Consulting and Clinical Psychology, 81 (1), 74-74 DOI: 10.1037/a0031156

Amir, N., Taylor, C., and Donohue, M. (2013). Correction to Amir et al. (2011). Journal of Consulting and Clinical Psychology, 81 (1), 112-112 DOI: 10.1037/a0031157

Unilaterally Raising the Scientific Standard

For years, I and others have been arguing that the current system of publishing science is broken. Publishing and peer-reviewing work only after the study's been conducted and the data analysed allows bad practices - such as selective publication of desirable findings, and running multiple statistical tests to find positive results - to run rampant.

So I was extremely interested when I received an email from Jona Sassenhagen, of the University of Marburg, with subject line: Unilaterally raising the standard.

Sassenhagen explained that he's chose to pre-register a neuroscience study on a public database, the German Clinical Trials Register (DRKS).

His project, Alignment of Late Positive ERP Components to Linguistic Deviations ("P600"), is designed to use EEG to test whether the brain generates a distinct electrical response - the P600 - in response to seeing grammatical errors. The background here is that the P600 certainly exists, but people disagree on whether it's specific to language; Sassenhagen hopes to find out.

By publicly announcing the methods he'll use before collecting any data, Sassenhagen has, in my view, taken a brave and important step towards a better kind of science.

Already, most journals require trials of medical treatments to be publicly pre-registered, and the DRKS is one such registry. This study, however, is 'pure' neuroscience with nothing clinical about it, so it doesn't need to be registered - Sassenhagen just did it voluntarily.

Further, I should point out that he offered to pre-register his data analysis pipeline too by sending it to me. Unfortunately, I didn't reply to the email in time... but that was purely my fault.

I very much hope and expect that others will follow in his footsteps. Unilaterally adopting preregistration is one of the ways that I've argued reform could get started. As I said:

This would, at least at first, place these adopters at an objective disadvantage. However, by voluntarily accepting such a disadvantage, it might be hoped that such actors would gain acclaim as more trustworthy than non-adopters.

Pre-registration puts you at a disadvantage - insofar as it limits your ability to use bad practice to fish for positive results. It means you can't cheat, essentially, which is a handicap if everyone else can.

I don't know if this is the first time anyone's opted in to registering a pure neuroscience study, but it's certainly the first case I know of it being done for an entirely new experiment.

There have, however, recently been many pre-registered attempts to replicate previously published results e.g. the Reproducibility of Psychological Science; the 'Precognition' Replications; and an upcoming special issue of Frontiers in Cognition.

Replications are good, registered ones doubly so - but they're not enough to fix bad practice on their own. To do that we need to work on the source, original scientific research.