Neuroskeptic Has Moved

After four years, Neuroskeptic is no longer an indie blog - I've moved to Discover Magazine blogs.

The new address is here: http://blogs.discovermagazine.com/neuroskeptic/

So please stop using this blogspot site; don't link to it, don't visit it, etc. All the posts are on the new one. The new RSS feed is: http://feeds.feedburner.com/neuro-skeptic/
Comments on this blog are now turned off but all old comments have been transferred to the new site, and you can comment there.

Neuroskeptic Is Moving

Over the next few days, Neuroskeptic will be moving to Discover Magazine. Which is very exciting.

This will mean no new posts for at least a week, while we sort out the technical issues of the transition, and I've also turned comments off - all existing comments will be moved over to the new blog, however.

For more updates you can follow me on Twitter...

See you on the other side soon!

Still 'Profiteering From Anxiety'

Late last year, the excellent Neurobonkers blog covered a case of 'Profiteering from anxiety'.

It seems one Nader Amir has applied for a patent on the psychological technique of 'Attentional Retraining', a method designed to treat anxiety and other emotional problems by conditioning the mind to unconsciously pay more attention to positive things and ignore unpleasant stuff.

For just $139.99, you can have a crack at modifying your unconscious with the help of Amir's Cognitive Retraining Technologies.

It's a clever idea... but hardly a new one. As Neurobonkers said, research on these kinds of methods had been going on for years before Amir came on the scene. In a comment, Prof. Colin MacLeod (who's been researching this stuff for over 20 years) argued that "I do not believe that a US patent granted to Prof Amir for the attentional bias modification approach would withstand challenge."

Well, in an interesting turn of events, Amir has issued just Corrections (1,2) to two of his papers. Both of the articles reported that retraining was an effective treatment for anxiety; but in both cases he now reveals that there was

an error...in the article a disclosure should have been noted that Nader Amir is the co-founder of a company that markets anxiety relief products.

Omitting to declare a conflict of interest... how unfortunate.

Still, it's an easy mistake to make: when you're focused on doing unbiased, objective, original research, as Amir doubtless was, such mundane matters are the last thing you tend to pay attention to.

Amir, N., and Taylor, C. (2013). Correction to Amir and Taylor (2012). Journal of Consulting and Clinical Psychology, 81 (1), 74-74 DOI: 10.1037/a0031156

Amir, N., Taylor, C., and Donohue, M. (2013). Correction to Amir et al. (2011). Journal of Consulting and Clinical Psychology, 81 (1), 112-112 DOI: 10.1037/a0031157

Unilaterally Raising the Scientific Standard

For years, I and others have been arguing that the current system of publishing science is broken. Publishing and peer-reviewing work only after the study's been conducted and the data analysed allows bad practices - such as selective publication of desirable findings, and running multiple statistical tests to find positive results - to run rampant.

So I was extremely interested when I received an email from Jona Sassenhagen, of the University of Marburg, with subject line: Unilaterally raising the standard.

Sassenhagen explained that he's chose to pre-register a neuroscience study on a public database, the German Clinical Trials Register (DRKS).

His project, Alignment of Late Positive ERP Components to Linguistic Deviations ("P600"), is designed to use EEG to test whether the brain generates a distinct electrical response - the P600 - in response to seeing grammatical errors. The background here is that the P600 certainly exists, but people disagree on whether it's specific to language; Sassenhagen hopes to find out.

By publicly announcing the methods he'll use before collecting any data, Sassenhagen has, in my view, taken a brave and important step towards a better kind of science.

Already, most journals require trials of medical treatments to be publicly pre-registered, and the DRKS is one such registry. This study, however, is 'pure' neuroscience with nothing clinical about it, so it doesn't need to be registered - Sassenhagen just did it voluntarily.

Further, I should point out that he offered to pre-register his data analysis pipeline too by sending it to me. Unfortunately, I didn't reply to the email in time... but that was purely my fault.

I very much hope and expect that others will follow in his footsteps. Unilaterally adopting preregistration is one of the ways that I've argued reform could get started. As I said:

This would, at least at first, place these adopters at an objective disadvantage. However, by voluntarily accepting such a disadvantage, it might be hoped that such actors would gain acclaim as more trustworthy than non-adopters.

Pre-registration puts you at a disadvantage - insofar as it limits your ability to use bad practice to fish for positive results. It means you can't cheat, essentially, which is a handicap if everyone else can.

I don't know if this is the first time anyone's opted in to registering a pure neuroscience study, but it's certainly the first case I know of it being done for an entirely new experiment.

There have, however, recently been many pre-registered attempts to replicate previously published results e.g. the Reproducibility of Psychological Science; the 'Precognition' Replications; and an upcoming special issue of Frontiers in Cognition.

Replications are good, registered ones doubly so - but they're not enough to fix bad practice on their own. To do that we need to work on the source, original scientific research.

Language That Is Person-First

An editorial in the Canadian Medical Association Journal by Roger Collier highlights the problem of Person-first language: Laudable cause, horrible prose

Person-first language (or language that is person-first, as it prefers to be known) is the nice idea that rather than calling someone, say, "blind", we should call them "a person who is blind", so as to remind everyone that they're not defined by their blindness but are a person first... clever, eh?

No. For one thing, it's just bad English. As Collier puts it: "There’s a reason Ernest Hemingway didn’t call his novel The Person Who Was Male and Advanced in Years and the Sea."

He goes on to quote linguist Helena Halmari who highlights a number of problems with the approach:

In English, emphasis naturally occurs at the end of sentences... so by pushing mention of a disability or disease deeper into a sentence, adherents to person-first language may actually be adding stress to those words. “What you have at the end of a sentence is the new information that gets the most attention,” says Halmari.

Worse yet...

Tucking the disability behind the noun may contribute to stigma rather than reduce it. After all, most adjectives with positive connotations precede nouns. We do not typically say a “person who is beautiful,” for instance, or a “person who is intelligent.” Sticking a word in the shadow of a noun can create the impression that there is something inherently wrong with it - that it should be hidden.

As a 'person with mental illness', I entirely agree. I am a man, a neuroscientist, a blogger; I'm not ashamed of those things, so I don't feel the need to erect linguistic fences between them and my person. I am also a psychiatric patient, a depressive, mentally ill; I'm not ashamed of that, either, and I resent the implication - however well-intentioned - that I should.

To me that's the really troubling part of this: the should aspect. The only reason you should not call someone something, is because they ask you not to.

Person-first advocates claim to be speaking on behalf of the 'group' who are harmed and offended by the current use of language - but who gave them that right? They don't speak for me, or anyone but themselves. I don't see 'the mentally ill' as a group at all, but even if it is one, they're certainly not our  elected representatives.

So non-person-first language doesn't offend me. In fact, I'm more worried by the idea that people will assume that, because I'm mentally ill, I want them to use person-first language. Now that's offensive.

Another Scuffle In The Coma Ward

It's not been a good few weeks for Adrian Owen and his team of Canadian neurologists.

Over the past few years, Owen's made numerous waves, thanks to his claim that some patients thought to be in a vegetative state may, in fact, be at least somewhat conscious, and able to respond to commands. Remarkable if true, but not everyone's convinced.

A few weeks ago, Owen et al were criticized over their appearance in a British TV program about their use of fMRI to measure brain activity in coma patients. Now, they're under fire from a second group of critics over a different project.

The new bone of contention is a paper published in 2011 called Bedside detection of awareness in the vegetative state. In this report, Owen and colleagues presented EEG results that, they said, show that some vegetative patients are able to understand speech.

In this study, healthy controls and patients were asked to imagine performing two different actions: moving their hand, or their toe. Owen et al found that it was possible to distinguish between the 'hand' and 'toe'-related patterns of brain electrical activity. This was true of most healthy control subjects, as expected, but also of some - not all - patients in a 'vegetative' state.

The skeptics aren't convinced, however. They reanalyzed the raw EEG data and claim that it just doesn't prove anything.

This image shows that in a healthy control, EEG activity was "clean" and generally normal. However in the coma patient, the data's a mess. It's dominated by large slow delta waves - in healthy people, you only see those during deep sleep - and there's also a lot of muscle artefacts which can be seen as 'thickening' of the lines.

These don't come from the brain at all, they're just muscle twitches. Crucially, the location and power of these twitches varied over time (as muscle spikes often do).

This wouldn't necessarily be a problem, the critics say, except that the statistics used by Owen et al didn't control for slow variations over time i.e. of correlations between consecutive trials (non-independence). If you do take account of these, there's no statistically significant evidence that you can distinguish the EEG associated with 'hand' vs 'toe' in any patients.

However, in their reply, Owen's team say that:

their reanalysis only pushes two of our three positive patients to just beyond the widely accepted p=0.05 threshold for significance - to p=0.06 and p=0·09, respectively. To dismiss the third patient, whose data remain significant, they state that the statistical threshold for accepting command-following should be adjusted for multiple comparisons... but we know of no groups in this field who routinely use such a conservative correction with patient data, including the critics themselves.

I have to say that, statistical arguments aside, the EEGs from the patients just don't look very reliable, largely because of those pesky muscle spikes. A new method for removing these annoyances has just been proposed... I wonder if that could help settle this?

Goldfine, A., Bardin, J., Noirhomme, Q., Fins, J., Schiff, N., and Victor, J. (2013). Reanalysis of "Bedside detection of awareness in the vegetative state: a cohort study" The Lancet, 381 (9863), 289-291 DOI: 10.1016/S0140-6736(13)60125-7

Is This How Memory Works?

We know quite a bit about how long-term memory is formed in the brain - it's all about strengthening of synaptic connections between neurons. But what about remembering something over the course of just a few seconds? Like how you (hopefully) still recall what that last sentence as about?

Short-term memory is formed and lost far too quickly for it to be explained by any (known) kind of synaptic plasticity. So how does it work? British mathematicians Samuel Johnson and colleagues say they have the answer: Robust Short-Term Memory without Synaptic Learning.

They write:

The mechanism, which we call Cluster Reverberation (CR), is very simple. If neurons in a group are more densely connected to each other than to the rest of the network, either because they form a module or because the network is significantly clustered, they will tend to retain the activity of the group: when they are all initially firing, they each continue to receive many action potentials and so go on firing.

The idea is that a neural network will naturally exhibit short-term memory - i.e. a pattern of electrical activity will tend to be maintained over time - so long as neurons are wired up in the form of clusters of cells mostly connected to their neighbours:

The cells within a cluster (or module) are all connected to each other, so once a module becomes active, it will stay active as the cells stimulate each other.

Why, you might ask, are the clusters necessary? Couldn't each individual cell have a memory - a tendency for its activity level to be 'sticky' over time, so that it kept firing even after it had stopped receiving input?

The authors say that even 'sticky' cells couldn't store memory effectively, because we know that the firing pattern of any individual cell is subject to a lot of random variation. If all of the cells were interconnected, this noise would quickly erase the signal. Clustering overcomes this problem.

But how could a neural clustering system develop in the first place? And how would the brain ensure that the clusters were 'useful' groups, rather than just being a bunch of different neurons doing entirely different things? Here's the clever bit:

If an initially homogeneous (i.e., neither modular nor clustered) area of brain tissue were repeatedly stimulated with different patterns... then synaptic plasticity mechanisms might be expected to alter the network structure in such a way that synapses within each of the imposed modules would all tend to become strengthened.

In other words, even if the brain started out life with a random pattern of connections, everyday experience (e.g. sensory input) could create a modular structure of just the right kind to allow short-term memory. Incidentally, such a 'modular' network would also be one of those famous small-world networks.

It strikes me as a very elegant model. But it is just a model, and neuroscience has a lot of those; as always, it awaits experimental proof.

One possible implication of this idea, it seems to me, is that short-term memory ought to be pretty conservative, in the sense that it could only store reactivations of existing neural circuits, rather than entirely new patterns of activity. Might it be possible to test that...?

Johnson S, Marro J, and Torres JJ (2013). Robust Short-Term Memory without Synaptic Learning. PloS ONE, 8 (1) PMID: 23349664

Is Medical Science Really 86% True?

The idea that Most Published Research Findings Are False rocked the world of science when it was proposed in 2005. Since then, however, it's become widely accepted - at least with respect to many kinds of studies in biology, genetics, medicine and psychology.

Now, however, a new analysis from Jager and Leek says things are nowhere near as bad after all: only 14% of the medical literature is wrong, not half of it. Phew!

But is this conclusion... falsely positive?

I'm skeptical of this result for two separate reasons. First off, I have problems with the sample of the literature they used: it seems likely to contain only the 'best' results. This is because the authors:

• only considered the creme-de-la-creme of top-ranked medical journals, which may be more reliable than others.
• only looked at the Abstracts of the papers, which generally contain the best results in the paper.
• only included the just over 5000 statistically significant p-values present in the 75,000 Abstracts published. Those papers that put their p-values up front might be more reliable than those that bury them deep in the Results.

In other words, even if it's true that only 14% of the results in these Abstracts were false, the proportion in the medical literature as a whole might be much higher.

Secondly, I have doubts about the statistics. Jager and Leek estimated the proportion of false positive p values, by assuming that true p-values tend to be low: not just below the arbitrary 0.05 cutoff, but well below it.

It turns out that p-values in these Abstracts strongly cluster around 0, and the conclusion is that most of them are real:

But this depends on the crucial assumption that false-positive p values are different from real ones, and equally likely to be anywhere from 0 to 0.05.

"if we consider only the P-­values that are less than 0.05, the P-­values for false positives must be distributed uniformly between 0 and 0.05."

The statement is true in theory - by definition, p values should behave in that way assuming the null hypothesis is true. In theory.

But... we have no way of knowing if it's true in practice. It might well not be.

For example, authors tend to put their best p-values in the Abstract. If they have several significant findings below 0.05, they'll likely put the lowest one up front. This works for both true and false positives: if you get p=0.01 and p=0.05, you'll probably highlight the 0.01. Therefore, false positive p values in Abstracts might cluster low, just like true positives.

Alternatively, false p's could also cluster the other way, just below 0.05. This is because running lots of independent comparisons is not the only way to generate false positives. You can also take almost-significant p's and fudge them downwards, for example by excluding 'outliers', or running slightly different statistical tests. You won't get p=0.06 down to p=0.001 by doing that, but you can get it down to p=0.04.

In this dataset, there's no evidence that p's just below 0.05 were more common. However, in many other sets of scientific papers, clear evidence of such "p hacking" has been found. That reinforces my suspicion that this is an especially 'good' sample.

Anyway, those are just two examples of why false p's might be unevenly distributed; there are plenty of others: 'there are more bad scientific practices in heaven and earth, Horatio, than are dreamt of in your model...'

In summary, although I think the idea of modelling the distribution of true and false findings, and using these models to estimate the proportions of each in a sample, is promising, I think a lot more work is needed before we can be confident in the results of the approach.

How To Respond to Criticism

People argue. On the internet, especially. Here's some tips on how best to respond to criticism of your ideas or writing - in my experience (the fact that I've often failed to follow these rules myself is part of that experience.)

Be Nice

Aggressive and insulting responses are a sign of weakness, and readers know it. If you're confident in your position, you can afford to be nice, and it makes your whole case look more convincing. Quite apart from the fact that it's just, well, nice.

Don't call out people for not being nice, though. The Three V's - "vitriolic", "virulent" and "violent" - seem to be especially common complaints. The trouble is that just as remarking on someone's faux pas is, itself, a faux pas, proclaiming that your opponent is using nasty language lowers the tone of your response.

It's natural to feel hurt by insults, but keep your feelings to yourself. Even if the criticism really is appallingly vicious, let it speak for itself: just slip a quote of the worst bits into your response, by way of making a separate point, and don't lower yourself by commenting on it.

Complimenting critics shows strength. It shows that you're confident that, despite the praise you're heaping on them, you're still right. So be generous. It only works if it seems sincere, though, so no outright brown-nosing.

Be Fresh

Don't just defend the ground you've already occupied - take the offensive (without being offensive). Bring new arguments to the table. New facts are always good - if a critic tries to debunk an example you used to prove a point, don't bother to quibble with them: produce three more.

Make your response readable. A reply is a piece of writing like any other, and it should be as concise and as clear as possible. Exhaustive replies are counterproductive; they're unlikely to be read. Just identify the key criticisms, and respond to those.

Stick to the point. Readers want you to engage with the issues. You may feel that you know all about your detractors' beliefs, character, motives and so forth, and that these are interesting. They rarely are.

Be Right

Often forgotten, this one.

If you're not sure whether you're right, find out. Take your time. If you have to say something right now, say you're working on it. Better a late reply than a bad reply.

If you're wrong, admit it. People will forgive an honest mistake, if you hold up your hands and ask them to. A reputation for admitting your mistakes and correcting your views is actually a point in your favor

If you've done something wrong, apologize. And nothing more. Don't try and justify yourself; it never works. Don't try and get people to pity you; it'll ensure no-one does. Just say sorry, and then keep quiet until the whole thing cools down.

How (Not) To Fix Social Psychology

British psychologist David Shanks has commented on the Diedrik Stapel affair and other recent scandals that have rocked the field of social psychology: Unconscious track to disciplinary train wreck,

Lots of people are chipping in on this debate for the first time at the moment, but peoples' initial reactions often fall prey to misunderstandings that can stand in the way of meaningful reform - misunderstandings that more considered analysis has exposed.

For example, Shanks writes:

[despite claims that] social psychology is no more prone to fraud than any other discipline, but outright fraud is not the major problem: the biggest concern is sloppy research practice, such as running several experiments and only reporting the ones that work.

It's true that fraud is not the major issue, as I and many others have said. But bad practice, such as p-value fishing, is in no way "sloppy" as Shanks says. Running multiple experiments to get a positive results is a sensible and effective strategy for getting positive results; that's why so many people do it. And so long as scientists are required to get such findings to get publications and grants, it will continue.

Behavior is the product of rewards and punishments, as a great psychologist said. We need to change the reinforcement schedule, not berate the rats for pressing the lever.

Earlier, Shanks writes that evidence of unconscious influences on human behaviour - a popular topic in Stapel's work and in social psychology generally -

is easily obtained because it usually rests on null results, namely finding that people's reports about (and hence awareness of) the causes of their behaviour fail to acknowledge the relevant cues. Null results are easily obtained if one's methods are poor.

Thus journals have in recent years published extraordinary reports of unconscious social influences on behaviour, including claims that people are more likely to take a cleansing wipe at the end of an experiment in which they are induced to recall an immoral act [etc]...

...failures to replicate the effects described above have been reported, though often papers reporting such failures are rejected out of hand by the journals that published the initial studies. I await with interest the outcome of efforts to replicate the recent claim that touching a teddy bear makes lonely people more sociable.

Here Shanks first says that null results can easily result from poorly-conducted experiments, and then criticizes journals for not publishing null results that represent failures to replicate prior claims! But null replications are very often rejected because a reviewer says, like Shanks, "This replication was just poorly-conducted, it doesn't count." Shanks (unconsciously no doubt) replicates the problem in his article.

So what to do? Again, it's a systemic problem. So long as we have peer-reviewed scientific journals, and the peer-review takes place after the data are collected, it will be open to reviewers to spike results they don't like - generally although not always null ones. If reviewers had to judge the quality of a study before they knew what it was going to find, as I've suggested, this problem would be solved.

Other people have great ideas for fixing science of their own. The problem is structural, not a failing on the part of individual scientists, and not limited to social psychology.

A Scuffle In The Coma Ward

A couple of months ago, the BBC TV show Panorama covered the work of a team of neurologists (led by Prof. Adrian Owen) who are pioneering the use of fMRI scanning to measure brain activity in coma patients.

The startling claim is that some people who have been considered entirely unconscious for years, are actually able to understand speech and respond to requests - not by body movements, but purely on the level of brain activation.

However, not everyone was impressed. A group of doctors swiftly wrote a critical response, published in the British Medical Journal as fMRI for vegetative and minimally conscious states: A more balanced perspective

The Panorama programme... failed to distinguish clearly between vegetative vs. minimally conscious states, and gave the impression that 20% of patients in a vegetative state show cognitive responses on fMRI.

There are important differences between the two states. Patients in a vegetative state have no discernible awareness of self and no cognitive interaction with their environment. Patients in a minimally conscious state show evidence of interaction through behaviours...

The programme presented two patients said to be in a “vegetative state” who showed evidence of cognitive interaction on assessment using fMRI but the clinical methods used for the original diagnosis were not stated. In both cases, family members clearly reported that the patient made positive but inconsistent behavioural responses to questions... one of these patients was filmed responding to a question from his mother by raising his thumb and the other seemed to turn his head purposefully.

So Panorama stands accused of passing off patients who were really minimally conscious, as being in a vegetative state. To see signs of understanding on brain scans from the latter would be truly amazing because it would be the first evidence that they weren't, well, vegetative.

However if they were 'merely' minimally conscious patients, it's not as interesting, because we already knew they were capable of making responses.

Now the Panorama team - and Professor Owen - have replied in a BMJ piece of their own. Given that they're charged with  misleading journalism and sloppy medicine, they're understandably a bit snarky:

Just by viewing this one hour documentary the authors felt able to discern that both the patients “said to be in a vegetative state” are “probably” minimally conscious... One of these patients, Scott, has had the same neurologist for more than a decade. Professor Young, who appeared in the film, made it clear that Scott had appeared vegetative in every assessment...

The fact that these authors took Scott’s fleeting movement, shown in the programme, to indicate a purposeful (“minimally conscious”) response shows why it is so important that the diagnosis is made in person, by an experienced neurologist, using internationally agreed criteria.

In other words, they were vegetative, and the critics who said otherwise, on the basis of some TV footage, were being silly.

In other words...it's on.

Turner-Stokes L, Kitzinger J, Gill-Thwaites H, Playford ED, Wade D, Allanson J, Pickard J, & Royal College of Physicians' Prolonged Disorders of Consciousness Guidelines Development Group (2012). fMRI for vegetative and minimally conscious states. BMJ (Clinical research ed.), 345 PMID: 23190911

Walsh F, Simmonds F, Young GB, & Owen AM (2013). Panorama responds to editorial on fMRI for vegetative and minimally conscious states. BMJ (Clinical research ed.), 346 PMID: 23298817

Drunk Rats Could Overturn Neurological Orthodoxy

A form of brain abnormality long regarded as permanent is, in fact, sometimes reversible, according to an unassuming little paper with big implications.

Here's the key data: some rats were given a lot of alcohol for four days (the "binge"), and then allowed to sober up for a week. Before, during and after their rodent Spring Break, they had brain scans. And these revealed something remarkable - the size of the rats' lateral ventricles increased during the binge, but later returned to normal.

Control rats, given lots of sugar instead of alcohol, did not show these changes.

This is really pretty surprising. The ventricles are simply fluid-filled holes in the brain. Increased ventricular size is generally regarded as a sign that the brain is shrinking - less brain, bigger holes - and if the brain is shrinking that must be because cells are dying or at least getting smaller. So bigger ventricles is bad.

Or so we thought... but this study shows that it might not always be true: alcohol reversibly increases ventricular volume over a timescale of days. It does so, the authors say, essentially by drying brain tissue out; like most things, if you dry the brain out, it gets smaller (and the ventricles get bigger) but when the water comes back to the tissues, it expands again.

As you can see here in Figure 2...

Maybe. I admit that just eyeballing this, it looks more like the ventricles are getting brighter, rather than bigger, but I'm not familiar with the details of water scanning. Maybe some readers will know more about it.

If it's true, this is big - maybe it's not just high doses of alcohol that does this. Maybe other drugs or factors can shrink or expand, the ventricles, or even other areas, purely by acting on tissue water regulation, rather than by anything more 'interesting'.

Take the various claims that some psychiatric drugs boost brain volume while others decrease it, just for starters...could they be headed for a watery grave?

Of course, this is in mice - and it might not translate to humans... we need to find out, and I for one am keen to apply for a grant. Here's my draft:

Participants: 8 healthy-livered neuroscientists.
Materials: 1 MRI scanner, 1 crate Jack Daniels.
Methods: Subjects will confer to pick a Designated Operator, who will remain sober. If no volunteers for this role are forthcoming, selection will be randomized by Bottle Spinning. All other participants will consume Jack Daniels ad libitum, and take turns being scanned. Once all Jack Daniels is depleted, participants will continue to be scanned until fully sobered up (defined as when they can successfully spell "amygdalohippocampal").
Instructions to Participants: i) what happens in the magnet, stays in the magnet. ii) If you 'dirty' the scanner, you clean it up. iii) Bottle caps are not MRI safe!

Er... seriously though, someone should check.

Zahr NM, Mayer D, Rohlfing T, Orduna J, Luong R, Sullivan EV, and Pfefferbaum A (2013). A mechanism of rapidly reversible cerebral ventricular enlargement independent of tissue atrophy. Neuropsychopharmacology  PMID: 23306181

DSM-5: A Ruse By Any Other Name...

In psychiatry, "a rose is a rose is a rose" as Gertrude Stein put it. That's according to an editorial in the American Journal of Psychiatry called: The Initial Field Trials of DSM-5: New Blooms and Old Thorns.

Like the authors, I was searching for some petal-based puns to start this piece off, but then I found this "flower with an uncanny resemblance to a MONKEY" which I think does the job quite nicely:

Anyway, the editorial is about the upcoming, controversial fifth revision to the Diagnostic and Statistical Manual (DSM) of the American Psychiatric Association (APA).

A great deal has been written about the DSM-5 over the past few years, as "the rough beast, its hour come round at last / Slouches towards Bethlehem to be born" (see, I can reference early-20th-century poetry too).

But now the talk has moved into a new phase, because the results of the DSM-5 'field trials' are finally out. In these studies, the reliability of the new diagnostic criteria for different psychiatric disorders was measured. The new editorial is a summary and discussion of the field trial data.

Two different psychiatrists assessed each patient, and the agreement between their diagnoses was calculated, as the kappa statistic, where 0 indicates no correlation at all and 1 is perfect.

It turns out that the reliabilities of most DSM-5 disorders were not very good. The majority were around 0.5, which is at best mediocre. These included such pillars of psychiatric diagnosis like schizophrenia, bipolar disorder, and alcoholism.

Others were worse. Depression, had a frankly crap kappa of 0.28, and the new 'Mixed Anxiety-Depressive Disorder' came in at -0.004 (sic). It was completely meaningless.

The American Journal editorial was written by a group of senior DSM-5 team members. I'm sure they wanted to write a triumphant presentation of their work, but in fact the tone is subdued, even apologetic in places:

As for most new endeavours, the end results are mixed, with both positive and disappointing findings...Experienced clinicians have severe reservations about the proposed research diagnostic scheme for personality disorder...like its predecessors, DSM-5 does not accomplish all that it intended, but it marks continued progress for many patients for whom the benefits of diagnoses and treatment were previously unrealized.

Remember: this is the journal published by the organization responsible for the DSM and even they don't much like it.

But the real story is even worse. The previous editions of the DSM also conducted field trials. These trials had a system to describe different kappa values: for example, 0.6-0.8 was 'satisfactory'.

However, the new DSM-5 studies used a different, lower threshold. They simply moved the goalposts, deeming lower kappa values to be good. At one point, they wrote that values of above 0.8 would be 'miraculous' and above 0.6 a 'cause for celebration', yet this wasn't the view of previous DSM developers.

The indispensable 1boringoldman blog has a nice graphic showing the results of the DSM-5 trials, with the kappas graded according to the old vs. the new criteria. As you can see, the grass is greener on the new side.

The fact is that the DSM-5 field trial results are worse than the results from DSM-III, the 1980 version that's served mostly unchanged for 30 years (DSM-IV made fairly modest changes.) The reliabilities have got worse - despite the editorial's claims of 'continued progress'. It's true that the DSM-5 field trials were a lot bigger and conducted rather differently, but still, it's a serious warning sign.

Finally, there was great variability in the results between different hospitals - in other words the reliability scores were not, themselves, reliable. Some institutions achieved much higher kappa values than others, but it's anyone's guess how they managed to do so.

Still, there's great news: the DSM-5 is just a piece of paper (well, a big stack of them). Any psychiatrist is free to ignore it - as the creator of the more reliable DSM-IV (not III, oops) is now urging them to do.

Freedman R, Lewis DA, Michels R, Pine DS, Schultz SK, Tamminga CA, Gabbard GO, Gau SS, Javitt DC, Oquendo MA, Shrout PE, Vieta E, and Yager J (2013). The Initial Field Trials of DSM-5: New Blooms and Old Thorns. The American Journal of Psychiatry, 170 (1), 1-5 PMID: 23288382

Smart People Say They're Less Depressed

The questionable validity of self-report measures in psychiatry has been the topic of a few recent  posts here at Neuroskeptic.

Now an interesting new study looks at the question in issue from a new angle, asking: what kind of people report feeling more or less depressed? Korean researchers Kim and colleagues found that intelligence and personality variables were both linked to the tendency to self-rate depression more severely.

The study involved 100 patients who'd previously suffered from an episode of depression or mania and who, according to their psychiatrist, had now recovered and were back to normal. Kim et al looked to see what the patient thought about their mood, by getting them to complete the Beck Depression Inventory (BDI) self-report questionnaire.

This was compared to the clinican-administered HAMD scale (another Neuroskeptic favourite) which is meant to be independent of self report.

It turns out that the BDI and HAMD scores were only weakly correlated - with a coefficient of just r=0.32. That's really not very good considering that, in theory, they both measure the same thing: 'depression'. Many people reported being considerably depressed when their clinicians rated them as fine.

But more interestingly, certain characteristics of the patients were correlated with their self-report/clinician-rating discrepancy. Specifically, patients with a lower IQ, who were more impulsive, and less conscientious, tended to self-report more severe depression.

Now, the uncharitable interpretation of these people is that they were just too sloppy to complete the form properly... the uncharitable interpretation of the psychiatrists is that it's their fault for underestimating depression in people less inclined to express themselves in 'the right way'. There's no way to know.

Either way, it's a serious problem because it shows that self-report and observer-report measures of depression aren't just poorly correlated, they're actually measuring different things for different people.

It could be even worse than it appears because the HAMD, although supposedly not a self-report measure, does in fact heavily rely on the patient's cooperation. So a 100% clinician-rated scale might be even further removed from self-report.

Kim EY, Hwang SS, Lee NY, Kim SH, Lee HJ, Kim YS, and Ahn YM (2012). Intelligence, temperament, and personality are related to over- or under-reporting of affective symptoms by patients with euthymic mood disorder. Journal of affective disorders PMID: 23270973

Artwork During Recovery From Encephalitis

I recently wrote about anti-NMDA receptor encephalitis, a neurological disorder that often manifests with psychiatric symptoms, such as depression and hallucinations.

The latest American Journal of Psychiatry features a strange series of four drawings made by a 15 year old girl during an episode of the disease, which presented as psychotic symptoms but later progressed to severe insomnia and epilepsy before it was diagnosed and treated.

"As she gradually recovered we asked her to draw something. She did not know what to draw, so we suggested an animal, such as a dog, but she did not know how to start.

When we told her that a dog has four legs, a tail, two ears, two eyes, and a mouth, she drew an abstract figure that consisted of a head with four legs (A). Her next drawing, of a cat, looked exactly the same, apparently since they share the same basic features.

Two weeks later the dog now looked more recognizable but like a human, standing upright, with two arms and four legs...All body parts were listed beneath the figure in the same color as they were drawn (B).

Two months after the patient was transferred to a local rehabilitation center, the cat was catlike for the first time; it had four legs, was normally proportioned, and was correctly positioned. Colors were used adequately. However, this drawing still looked like one by a primary school child instead of a 15- year-old girl (C).

Finally, after 5 months of rehabilitation her drawing had a normal composition. She still had the urge to write down what she drew, she did not encircle the figures anymore (D)."

Esseveld MM, van de Riet EH, Cuypers L, and Schieveld JN (2013). Drawings During Neuropsychiatric Recovery From Anti-NMDA Receptor Encephalitis. The American journal of psychiatry, 170 (1), 21-2 PMID: 23288386

Notorious Paedophile Reads Neuroskeptic

There's been controversy in the UK over an article published in the Guardian that's regarded as being pro-child abuse.

In particular, the newspaper has taken flak for quoting Tom O'Carroll, a self-confessed paedophile and advocate for the right of people to be one. Amongst other things he's written a book about Michael Jackson. At least this week, until the next one comes along, he is Britain's most notorious paedophile.

Now oddly enough, I recently had an encounter with O'Carroll, although I didn't know who he was at the time. Here's the tale...

A few weeks ago, I got an out-of-the-blue email from a Neuroskeptic reader (as happens often), from someone saying he'd tried to leave a comment on this post but it was rejected for being too long.

O'Carroll (for it was he) wanted to know whether I agreed with his reservations about some research on "Cerebral white matter deficiencies in pedophilic men", and specifically on this statement by the paper's author:

One must consider carefully whether the brain differences we detected [i.e. reduced white matter volume in particular areas] cause pedophilia or whether some aspect of being pedophilic caused the brain differences...

Although it is now known that certain brain structures respond to environmental stimulation, such as the motor cortex, there is no evidence that such stimulation causes any changes in the superior fronto-occipital fasciculus or right arcuate fasciculus (the brain regions in which pedophiles and nonpedophiles differ).

Moreover, the brain regions we identified are extremely large, and no previous research has ever found changes in such large regions of the brain. As an analogy, physical exercise will generally stimulate one’s muscle tissue to grow, but one would not grow an extra arm; neurological changes occur only in a very specific manner.

O'Carroll thought that this is a misleading analogy, because the brain could be plastic in ways we don't yet understand. I agreed. We just don't know enough about the brain, yet, to say that the observed white matter changes can't possibly be responses to experiences.

Several recent studies found that experience and learning can change adult human white matter structure. The changes observed were fairly small, but then these studies were fairly short, so they don't define the upper limit of what's possible over time.

These results are not without critics of their own, and I'm on the fence about whether white matter is plastic at all, but my point is, it's an open question. So comparing the idea of white matter plasticity to 'growing an extra arm' is overstatement - and it would be, whether the topic was paedophilia or anything else.

Now, as I said, I hadn't heard of Tom O'Carroll at the time, and I assumed he had a purely academic interest in the matter, as an piece of oversold neuroscience. But now, thanks to the Guardian drama, I realize that...

Britain's most notorious paedophile reads Neuroskeptic.

Hmm.

On that macabre note, I've often wondered whether James Eagan Holmes, the Aurora, Colorado Batman shooter, ever visited this blog. It's possible: he was a neuroscience undergraduate and later PhD student over the time Neuroskeptic's been going.

There have been 574 visits from Aurora, Colorado, where Holmes was doing his PhD, since 2008. I'll never know whether he was one of them, but the idea that he might have been is pretty creepy.

Flawed Statistics Make Almost Everyone's Brain "Abnormal"

A popular method for detecting abnormalities in the shape and size of individual brains is seriously flawed, and is almost guaranteed to find 'differences' even in normal people.

So say Italian neuroscientists Scarpazza and colleagues in an important new report: Very high false positive rates in single case Voxel Based Morphometry.

Voxel Based Morphometry (VBM) is a way of analyzing brain scans to detect structural differences. It's most commonly used to compare groups of brains to find average differences, but some neuroscientists have started using VBM to check for abnormalities in a single brain. Scarpazza et al list 34 pieces of research about that, including 13 since 2010.

So it would suck if there were a problem with individual VBM... but there is. This pic tells the tale:

The authors took 200 normal brains and compared each one of them in turn to a control group of 16 normal brains. Because all of them were healthy, the comparisons ought to show no significant differences.

The technique was set up so that, in theory, only 5% of the brains should have been wrongly labelled as containing an abnormality. But in fact, a full 93.5% of the normal brains gave at least one false positive.

So 5% is more like the rate of not being wrong. Oops.

The image shows that in some brain areas, almost 25% of the normal brains were branded as 'abnormal' just in that region alone - the hotter the colour, the higher the proportion of false 'hits'. The top row is for false reports of brain volume increases, while the bottom row is decreases; false 'increases' were more common.

So what's going wrong? It's not entirely clear and several factors are probably at play, but the authors say that the main issue is that VBM makes the assumption of statistical normality which doesn't in fact hold.

Either way, it's a serious problem, and Scarpazza et al point to one especially worrying implication: some people have proposed using single-subject VBM in a legal context, to reinforce insanity pleas by showing subtle 'brain abnormalities' not obvious to the naked eye. Yet if this paper's right, such evidence could be entirely meaningless, almost guaranteed to give a positive result.

P.S. Last time I posted about this kind of analysis flaw, the internet went crazy because they didn't understand it. So just to be clear, this is not a problem for clinical scans - the kind you'd get to check whether you have a brain tumour.

Scarpazza, C., Sartori, G., De Simone, M., and Mechelli, A. (2013). When the single matters more than the group: Very high false positive rates in single case Voxel Based Morphometry NeuroImage DOI: 10.1016/j.neuroimage.2012.12.045