DSM-5: A Ruse By Any Other Name...

In psychiatry, "a rose is a rose is a rose" as Gertrude Stein put it. That's according to an editorial in the American Journal of Psychiatry called: The Initial Field Trials of DSM-5: New Blooms and Old Thorns.

Like the authors, I was searching for some petal-based puns to start this piece off, but then I found this "flower with an uncanny resemblance to a MONKEY" which I think does the job quite nicely:

Anyway, the editorial is about the upcoming, controversial fifth revision to the Diagnostic and Statistical Manual (DSM) of the American Psychiatric Association (APA).

A great deal has been written about the DSM-5 over the past few years, as "the rough beast, its hour come round at last / Slouches towards Bethlehem to be born" (see, I can reference early-20th-century poetry too).

But now the talk has moved into a new phase, because the results of the DSM-5 'field trials' are finally out. In these studies, the reliability of the new diagnostic criteria for different psychiatric disorders was measured. The new editorial is a summary and discussion of the field trial data.

Two different psychiatrists assessed each patient, and the agreement between their diagnoses was calculated, as the kappa statistic, where 0 indicates no correlation at all and 1 is perfect.

It turns out that the reliabilities of most DSM-5 disorders were not very good. The majority were around 0.5, which is at best mediocre. These included such pillars of psychiatric diagnosis like schizophrenia, bipolar disorder, and alcoholism.

Others were worse. Depression, had a frankly crap kappa of 0.28, and the new 'Mixed Anxiety-Depressive Disorder' came in at -0.004 (sic). It was completely meaningless.

The American Journal editorial was written by a group of senior DSM-5 team members. I'm sure they wanted to write a triumphant presentation of their work, but in fact the tone is subdued, even apologetic in places:

As for most new endeavours, the end results are mixed, with both positive and disappointing findings...Experienced clinicians have severe reservations about the proposed research diagnostic scheme for personality disorder...like its predecessors, DSM-5 does not accomplish all that it intended, but it marks continued progress for many patients for whom the benefits of diagnoses and treatment were previously unrealized.

Remember: this is the journal published by the organization responsible for the DSM and even they don't much like it.

But the real story is even worse. The previous editions of the DSM also conducted field trials. These trials had a system to describe different kappa values: for example, 0.6-0.8 was 'satisfactory'.

However, the new DSM-5 studies used a different, lower threshold. They simply moved the goalposts, deeming lower kappa values to be good. At one point, they wrote that values of above 0.8 would be 'miraculous' and above 0.6 a 'cause for celebration', yet this wasn't the view of previous DSM developers.

The indispensable 1boringoldman blog has a nice graphic showing the results of the DSM-5 trials, with the kappas graded according to the old vs. the new criteria. As you can see, the grass is greener on the new side.

The fact is that the DSM-5 field trial results are worse than the results from DSM-III, the 1980 version that's served mostly unchanged for 30 years (DSM-IV made fairly modest changes.) The reliabilities have got worse - despite the editorial's claims of 'continued progress'. It's true that the DSM-5 field trials were a lot bigger and conducted rather differently, but still, it's a serious warning sign.

Finally, there was great variability in the results between different hospitals - in other words the reliability scores were not, themselves, reliable. Some institutions achieved much higher kappa values than others, but it's anyone's guess how they managed to do so.

Still, there's great news: the DSM-5 is just a piece of paper (well, a big stack of them). Any psychiatrist is free to ignore it - as the creator of the more reliable DSM-IV (not III, oops) is now urging them to do.

Freedman R, Lewis DA, Michels R, Pine DS, Schultz SK, Tamminga CA, Gabbard GO, Gau SS, Javitt DC, Oquendo MA, Shrout PE, Vieta E, and Yager J (2013). The Initial Field Trials of DSM-5: New Blooms and Old Thorns. The American Journal of Psychiatry, 170 (1), 1-5 PMID: 23288382

Smart People Say They're Less Depressed

The questionable validity of self-report measures in psychiatry has been the topic of a few recent  posts here at Neuroskeptic.

Now an interesting new study looks at the question in issue from a new angle, asking: what kind of people report feeling more or less depressed? Korean researchers Kim and colleagues found that intelligence and personality variables were both linked to the tendency to self-rate depression more severely.

The study involved 100 patients who'd previously suffered from an episode of depression or mania and who, according to their psychiatrist, had now recovered and were back to normal. Kim et al looked to see what the patient thought about their mood, by getting them to complete the Beck Depression Inventory (BDI) self-report questionnaire.

This was compared to the clinican-administered HAMD scale (another Neuroskeptic favourite) which is meant to be independent of self report.

It turns out that the BDI and HAMD scores were only weakly correlated - with a coefficient of just r=0.32. That's really not very good considering that, in theory, they both measure the same thing: 'depression'. Many people reported being considerably depressed when their clinicians rated them as fine.

But more interestingly, certain characteristics of the patients were correlated with their self-report/clinician-rating discrepancy. Specifically, patients with a lower IQ, who were more impulsive, and less conscientious, tended to self-report more severe depression.

Now, the uncharitable interpretation of these people is that they were just too sloppy to complete the form properly... the uncharitable interpretation of the psychiatrists is that it's their fault for underestimating depression in people less inclined to express themselves in 'the right way'. There's no way to know.

Either way, it's a serious problem because it shows that self-report and observer-report measures of depression aren't just poorly correlated, they're actually measuring different things for different people.

It could be even worse than it appears because the HAMD, although supposedly not a self-report measure, does in fact heavily rely on the patient's cooperation. So a 100% clinician-rated scale might be even further removed from self-report.

Kim EY, Hwang SS, Lee NY, Kim SH, Lee HJ, Kim YS, and Ahn YM (2012). Intelligence, temperament, and personality are related to over- or under-reporting of affective symptoms by patients with euthymic mood disorder. Journal of affective disorders PMID: 23270973

Artwork During Recovery From Encephalitis

I recently wrote about anti-NMDA receptor encephalitis, a neurological disorder that often manifests with psychiatric symptoms, such as depression and hallucinations.

The latest American Journal of Psychiatry features a strange series of four drawings made by a 15 year old girl during an episode of the disease, which presented as psychotic symptoms but later progressed to severe insomnia and epilepsy before it was diagnosed and treated.

"As she gradually recovered we asked her to draw something. She did not know what to draw, so we suggested an animal, such as a dog, but she did not know how to start.

When we told her that a dog has four legs, a tail, two ears, two eyes, and a mouth, she drew an abstract figure that consisted of a head with four legs (A). Her next drawing, of a cat, looked exactly the same, apparently since they share the same basic features.

Two weeks later the dog now looked more recognizable but like a human, standing upright, with two arms and four legs...All body parts were listed beneath the figure in the same color as they were drawn (B).

Two months after the patient was transferred to a local rehabilitation center, the cat was catlike for the first time; it had four legs, was normally proportioned, and was correctly positioned. Colors were used adequately. However, this drawing still looked like one by a primary school child instead of a 15- year-old girl (C).

Finally, after 5 months of rehabilitation her drawing had a normal composition. She still had the urge to write down what she drew, she did not encircle the figures anymore (D)."

Esseveld MM, van de Riet EH, Cuypers L, and Schieveld JN (2013). Drawings During Neuropsychiatric Recovery From Anti-NMDA Receptor Encephalitis. The American journal of psychiatry, 170 (1), 21-2 PMID: 23288386

Notorious Paedophile Reads Neuroskeptic

There's been controversy in the UK over an article published in the Guardian that's regarded as being pro-child abuse.

In particular, the newspaper has taken flak for quoting Tom O'Carroll, a self-confessed paedophile and advocate for the right of people to be one. Amongst other things he's written a book about Michael Jackson. At least this week, until the next one comes along, he is Britain's most notorious paedophile.

Now oddly enough, I recently had an encounter with O'Carroll, although I didn't know who he was at the time. Here's the tale...

A few weeks ago, I got an out-of-the-blue email from a Neuroskeptic reader (as happens often), from someone saying he'd tried to leave a comment on this post but it was rejected for being too long.

O'Carroll (for it was he) wanted to know whether I agreed with his reservations about some research on "Cerebral white matter deficiencies in pedophilic men", and specifically on this statement by the paper's author:

One must consider carefully whether the brain differences we detected [i.e. reduced white matter volume in particular areas] cause pedophilia or whether some aspect of being pedophilic caused the brain differences...

Although it is now known that certain brain structures respond to environmental stimulation, such as the motor cortex, there is no evidence that such stimulation causes any changes in the superior fronto-occipital fasciculus or right arcuate fasciculus (the brain regions in which pedophiles and nonpedophiles differ).

Moreover, the brain regions we identified are extremely large, and no previous research has ever found changes in such large regions of the brain. As an analogy, physical exercise will generally stimulate one’s muscle tissue to grow, but one would not grow an extra arm; neurological changes occur only in a very specific manner.

O'Carroll thought that this is a misleading analogy, because the brain could be plastic in ways we don't yet understand. I agreed. We just don't know enough about the brain, yet, to say that the observed white matter changes can't possibly be responses to experiences.

Several recent studies found that experience and learning can change adult human white matter structure. The changes observed were fairly small, but then these studies were fairly short, so they don't define the upper limit of what's possible over time.

These results are not without critics of their own, and I'm on the fence about whether white matter is plastic at all, but my point is, it's an open question. So comparing the idea of white matter plasticity to 'growing an extra arm' is overstatement - and it would be, whether the topic was paedophilia or anything else.

Now, as I said, I hadn't heard of Tom O'Carroll at the time, and I assumed he had a purely academic interest in the matter, as an piece of oversold neuroscience. But now, thanks to the Guardian drama, I realize that...

Britain's most notorious paedophile reads Neuroskeptic.

Hmm.

On that macabre note, I've often wondered whether James Eagan Holmes, the Aurora, Colorado Batman shooter, ever visited this blog. It's possible: he was a neuroscience undergraduate and later PhD student over the time Neuroskeptic's been going.

There have been 574 visits from Aurora, Colorado, where Holmes was doing his PhD, since 2008. I'll never know whether he was one of them, but the idea that he might have been is pretty creepy.

Flawed Statistics Make Almost Everyone's Brain "Abnormal"

A popular method for detecting abnormalities in the shape and size of individual brains is seriously flawed, and is almost guaranteed to find 'differences' even in normal people.

So say Italian neuroscientists Scarpazza and colleagues in an important new report: Very high false positive rates in single case Voxel Based Morphometry.

Voxel Based Morphometry (VBM) is a way of analyzing brain scans to detect structural differences. It's most commonly used to compare groups of brains to find average differences, but some neuroscientists have started using VBM to check for abnormalities in a single brain. Scarpazza et al list 34 pieces of research about that, including 13 since 2010.

So it would suck if there were a problem with individual VBM... but there is. This pic tells the tale:

The authors took 200 normal brains and compared each one of them in turn to a control group of 16 normal brains. Because all of them were healthy, the comparisons ought to show no significant differences.

The technique was set up so that, in theory, only 5% of the brains should have been wrongly labelled as containing an abnormality. But in fact, a full 93.5% of the normal brains gave at least one false positive.

So 5% is more like the rate of not being wrong. Oops.

The image shows that in some brain areas, almost 25% of the normal brains were branded as 'abnormal' just in that region alone - the hotter the colour, the higher the proportion of false 'hits'. The top row is for false reports of brain volume increases, while the bottom row is decreases; false 'increases' were more common.

So what's going wrong? It's not entirely clear and several factors are probably at play, but the authors say that the main issue is that VBM makes the assumption of statistical normality which doesn't in fact hold.

Either way, it's a serious problem, and Scarpazza et al point to one especially worrying implication: some people have proposed using single-subject VBM in a legal context, to reinforce insanity pleas by showing subtle 'brain abnormalities' not obvious to the naked eye. Yet if this paper's right, such evidence could be entirely meaningless, almost guaranteed to give a positive result.

P.S. Last time I posted about this kind of analysis flaw, the internet went crazy because they didn't understand it. So just to be clear, this is not a problem for clinical scans - the kind you'd get to check whether you have a brain tumour.

Scarpazza, C., Sartori, G., De Simone, M., and Mechelli, A. (2013). When the single matters more than the group: Very high false positive rates in single case Voxel Based Morphometry NeuroImage DOI: 10.1016/j.neuroimage.2012.12.045