There's been a lot of buzz and some scepticism about theNew brain scan to diagnose autismHere's a quick overview. Autism is believed to be a disorder of brain development. If so, it should be possible to diagnose it based on a brain scan. Unfortunately, it's not. You can't tell, from a scan, whether someone has autism or not. Not even if you're a world expert.
There are reports of various differences between autistic and non-autistic brains - a bit smaller here, a bit bigger there - but there's a lot of overlap. So at present, diagnosis of autism is purely based on symptoms.
Ecker et al, a team based at the Institute of Psychiatry in South London, made use of a mathematical technique called a Support Vector Machine (SVM) to try to spot differences that the naked eye can't. An SVM is a learning algorithm: you "teach" it to spot differences by showing it lots of examples. In this case, they showed it 20 autistic brains, and the brains of 20 healthy controls matched for age, gender, and IQ.
How does an SVM work? Imagine that there are two kinds of, say, fruit. Both are kind of round but A's are more spherical than B's. So you could draw a plot of sphere-ness, and find a line separating A and B:
An SVM is an automatic method of finding that line. How? It's complex, but fortunately you don't need to know (I don't). Of course, that's easy, but imagine that things got more tricky. As well as the variable of roundness, there's colour. Fruit B can be either spherical and dark, or non-spherical and light (maybe it's two different stages of ripeness).An SVM could do this easily too:
Now suppose that there's 1000 different variables, and you want to find the "line" - actually a 1000-dimensional "hyperplane" (a line is 2D, a plane is 3D, anything with 4D or more is a hyper-plane) - dividing the "space" of possibilities into two.For a human that's impossible, but not for an SVM. This is essentially what Ecker et al did. Each dimension of their "space" was the amount of grey matter at a particular point in the brain. So, they were training the SVM to distinguish between autistic brains and non-autistic brains, based on their shape, but in a much more complex way than a human could.
Did it work? Surprisingly well. Here's the end result (the multi-dimensional space has been helpfully compressed into 2D by the SVM):
It wasn't perfect, but the best approach, based on the cortical thickness in the left hemisphere, managed 90% accuracy, which is pretty awesome. Focussing on the headline 90% result is cherry-picking a bit, because using other variables, like cortical curvature, wasn't as good, but even the worst ones managed 70-85%, much better than chance (50%). Importantly, they also tried the system on 20 adults with ADHD, and it classified them as non-autistic. This shows that it's not just measuring "normality".*
Now the question everyone's asking: is this going to be used for diagnosis in the real world any time soon? The first thing to remember is that this is a scientific paper, and this result is first and foremost of research interest: it provides clues towards the biology, and ultimately the causes, of autism.
But let's suppose you're a clinician and you have someone who you suspect may have autism, but you're not sure. They're a tricky one, a borderline case. You use this system on their brain and it says they are autistic. Should that factor into your decision? It depends. The fact is that rather than an either-or result, the SVM returns a distance from the hyperplane for each brain. You can see this clearly in the plot above.
In my opinion, if you have a borderline case, and the machine says he's borderline, then that's not much help, and it doesn't matter if he's just over the line, or not quite over it. You already knew he was borderline.
But if the machine says that he's deep into the autism space, then I think that is something. It tells you that his brain is very typical of people with autism. Interestingly, Ecker et al found that distance from the hyperplane correlated with symptom severity for "social" and "communication" symptoms (though not "repetitive behaviours"). That's a pretty cool result because the SVM wasn't trained to do that, it was trained to decide on an either-or basis.
What needs to happen next? As it stands, this system only works for adults: it would fail for children or teenagers, because their brains are a very different size and shape. Exactly the same SVM approach could be used in younger age groups, though, so long as the patients and the controls were the same age.
We also need to make sure that the SVM can tell the difference between autism and other conditions; Ecker et al showed that it could distinguish autism from ADHD, but that's only one comparison and it might not be the hardest one: I would want to see it tested against things like epilepsy, mental retardation, and dyslexia as well.
Overall though, this is very exciting work, and certainly a cut above most "Brain Scans To Diagnose Mental Illness" studies that make it into the headlines.
Full Discloser: I know some of the researchers involved in this work.
Links: The same team had a paper out a few months back, using a slightly less sophisticated SVM approach, which managed 80% accuracy. I wrote about another application of SVMs previously: How To Read Minds. This study has been blogged about at The New Republic and Dormivigilia.
Ecker C, Marquand A, Mourão-Miranda J, Johnston P, Daly EM, Brammer MJ, Maltezos S, Murphy CM, Robertson D, Williams SC, & Murphy DG (2010). Describing the brain in autism in five dimensions--magnetic resonance imaging-assisted diagnosis of autism spectrum disorder using a multiparameter classification approach. Journal of Neuroscience, 30 (32), 10612-23 PMID: 20702694
26 comments:
I take my blogging hat off to you. A fine piece of work :)
"the SVM returns a distance from the hyperplane for each brain"
That line would just work so well mixed into an Eat Static track
Thank you for an excellent and easily understandable explanation of this study!
Autism falls along a broad spectrum from severe to mild (Asperger's). How well does this study's results generalize to various types of autism?
The results in the study are interesting but the reporting of it, including by the researchers is shameful. I was doing the stats myself until I realised the Guardian article you link to does a fine job. The 90% accuracy seems high but it is a horribly misleading number. As Carl Heneghan explains, the positive predictive value for this test is about 5%. This work should be reported and valued, but it is far-fetched and scientifically inappropriate to report this test as '90% accurate'. It almost seems that sensitivity is a statistic you would choose and describe that way only if you intentionally wanted to mislead.
"Dr Ecker said she hoped the findings might result in a widely available scan to test for autism." See, the study has all the caveats one would expect, then the headline and the quotes on the press release totally mislead. No wonder people look disappointed when I explain to them that we know not much about cognition and the brain and the popular science they have read is wrong.
Was it done entirely on boys? Because it would be interesting to see if it could be used to diagnose young women who have had difficulties that might be on the spectrum but whose behavioral issues are not perhaps as dramatic as those of,say, a brother with autism.
One of the great difficulties for girls on the spectrum is that there are fewer of them, and so less research on them, and they often present differently.
I also wonder if this might pick up family members who have a soft form of the disorder or who carry the gene, but are apparently typical. A concern here would be that a test could be used to discriminate against a person, exclude them from certain occupations or the like.
Bravo. But like all neuroscience research into psychiatric disorders, it means nothing. What are the treatment implications???
You neglect to mention that 16 out of 20 (80% of the subjects) had Asperger's syndrome rather than autism without a speech delay, so this would be limitation of the work in that it might not apply to autistics with a speech delay or more severely autistic. Also, there were no women in the experimental group and I don't think the control group either.
Also, I was wondering if they had used a control group of women what would happen. There are certainly differences (including probably cortical thickening) of women's versus men's brains, so a control group of healthy women might be diagnosed as autistic as well, based on this scan.
Also, the high percentage sensitivity was limited to the left hemisphere.
Thanks for the clear explanation of the methods.
To my mind the media reporting of this article is grossly misleading, and it seems that the MRC and the authors are largely responsible for dressing up an interesting study of brain correlates of autism as a story about diagnosis and screening. Presumably this was done to make it 'relevant' in some way.
If you read the article, the authors state re diagnostic potential:
"This would, however, require further extensive exploration in the clinical setting, particularly with regards to classifier specificity to ASD rather than neurodevelopmental conditions in general." (p. 10621).
Their caution is merited. Contrary to what you state in the blog, 21% of people with ADHD were diagnosed as autistic. And a similar number of controls with no diagnosis whatsoever also had an 'autistic brain'. No work has been done to demonstrate diagnostic potential in real clinical settings, where much of the difficulty is distinguishing autism from other neurodevelopmental disorders, especially language impairment.
However, in the MRC press release, we hear:
"Scientists funded by the Medical Research Council (MRC) have developed a pioneering new method of diagnosing autism in adults. For the first time, a quick brain scan that takes just 15 minutes can identify adults with autism with over 90% accuracy. The method could lead to the screening for autism spectrum disorders in children in the future."
http://www.mrc.ac.uk/Newspublications/News/MRC007083
This kind of hype is more what we expect from the Daily Mail than a responsible research funder. In addition to Carl Heneghan's excellent analysis which you have cited, further critiques can be found on my blog http://tiny.cc/toidc (in a piece written some 3 weeks *before* this article was published! – extrapolation from research studies to 'screening' claims is nothing new, alas), and also from the excellent NHS Choices website http://tiny.cc/yntbh.
Anonymous: The problem with Carl Heneghan's argument is that it assumes this would be used on everyone in the population. Most people however are obviously not autistic, so there would be no point in scanning them. Where it might be useful is in borderline cases.
Retriever: Yes, and I think distinguishing between autistic people and their relatives is going to be the hardest challenge faced by this kind of test.
On the other hand, we know that many people who have relatives with autism do have autistic traits, so you would hope that the SVM would pick up on that.
Ultimately what we want is a system which tells you your place on the spectrum of symptoms, based on your distance from some hyperplane. That's some way off, but I think this work is an important first step.
re "Ultimately what we want is a system which tells you your place on the spectrum of symptoms, based on your distance from some hyperplane."
Hmm. If you want to know your place on the spectrum of symptoms, then I suggest you evaluate the symptoms. This reductionist notion that you can predict symptoms accurately from a brain measure has to date proved unrealistic for most neurodevelopmental disorders; not only is there no one-to-one relation between symptom and brain, but you can also find the same anomalies in brains of people with very different conditions. I agree this study is promising but given the small N and many prior failures to find consistent brain markers of autism vs other conditions, we'd need much more evidence before regarding it as diagnostically useful.
My properly diagnosed autistic brain tells me it's a ludicrous study.
Using unreliable methods, testing a small group and finding things in common. And then concluding something from that.
AGW has better science.
The ADI-R doesn't work like a scale of autism "severity." It provides thresholds only.
There is now preliminary published work attempting make the ADOS work like a scale of autism "severity." It has had limited replication and doesn't extend to module 4. But no such work has yet been published re the ADI-R.
Most of the autistic sample in Ecker et al. was diagnosed by ADI-R only; only two were diagnosed by both ADI-R and ADOS (the standard in the literature). The cut-offs used were not stated.
Also, the autistic and nonautistic groups weren't matched on performance IQ.
Apart from being limited to males, this study (and its classifier) is limited to right-handed people.
In their related previous study, Ecker and colleagues used both ADI-R and ADOS for diagnosis, and took AQ scores as well. They found correlations (with distance from hyperplane) for the ADOS improperly used as a scale, and for the AQ (which is a scale, but there is some disagreement about what it is measuring), but no correlation with ADI-R (again, improperly used as a scale). The authors' explanation was that the ADI-R refers to early development:
"classification on the basis of brain morphometry in adulthood is predominantly driven by current symptoms as measured by ADOS rather than by past symptom severity during childhood as measured by the ADI."
Forgot to add... re cortical thickness, the crucial measure in this study, there's a relevant APS editorial from 2007.
Thanks Michelle, that confirms it then, since ADOS and ADI as diagnostic tests are entirely social cosntructs around what ought to be in Autism, the test tells us nothing new, doesn't validate that there even is such a thing as autism, only it proves that there is some neurological correllation with a particular set of social constructions.
But then if you go hunting for white balls in a pool of black ones that is what you will find, and if you go hunting for yellow balls in a pool of black and white ones that is what you will find.
They could just as well have jiggered the parameters of autism differently and still had the same findings because all human behavior has some neurological correlates and any mix of behaviours would have it's own distinct pattern of differences.
Oh well it will be back to the drawing board I suppose.
Didn't read the paper but I know something about SVM: A sample with 20 elements is very small, I assume that a human being can spot the difference too?
If we test the SVM's model with other 1000s with&without autism, what's the error rate?
srw,
No a human cannot spot the difference b/c this is a structural MRI where there are - not sure here - thousands of surface voxels, each of which has 5 different dimensions. Each subject has a TON of data.
Any test that shows a very strong difference between people with autism and the general population is suspect in itself, because the one thing we can say confidently is that the diagnostic criteria for autism are so fuzzy, not to say all-encompassing, that some overlap is inevitable.
DSM-IV, for example, takes a three from column A and two from column B approach, so that if you calculate the perms and combs there are some 50,000 different ways to be autistic, and that's a problem. It also requires you to say when something like communication impairment is greater than normal without having any reference for what's normal, and that's a problem. Being autistic, when you look at it, is diagnosed not against objective criteria but against type specimens, such as rainman. Being autistic is a lot more like "being like Hamlet" than it's like "being 1.8 metres tall', and that's the real problem.
Is it autism or something else? I'm assuming you can apply algorithms to anything, the difference being the input data. It seems self-defeating to collate autism data then test to see whether it is in fact autism.
Why on earth would you need a machine to detect autism?
Isn't it easier to detect behaviourally whether someone is autistic or not? I can only imagine such machines in the courtroom to plea beyond reasonable doubt whether someone is autistic or not, despite the fact him/her sitting in the box looks totally autistic.
If it's a matrix reflecting the input data then it should be 100% accurate. 90% is not on. That's like my calculator being correct 90% of the time. The mathematical constructs concerning 11 dimensions is supposed to be perfect.
These algorithms have also detected 2012 to be the end of the world based on input data. Can someone please tell me if this is 90% correct? Are the matrices 90% certain the world will end in 2012? I need to know.
If someone is borderline why would you want to know? What does borderline even mean? Like a mixed race person choose whether you're latino or anglo. Get some counselling, live your life then die like everybody else. I should be an autism counsellor.
Also maybe they chose to neglect the female cohort because they were masculine? I read somewhere female autistic brains were kind of masculine? That would raise some ethical concerns.
"." - That's an important point, and it's possible that these patients were all of a particular "type" (i.e. 16 out of 20 had no language delay, making them "Asperger's" as opposed to "HFA") and that it wouldn't detect others.
Ultimately we just need more patients. This is where some kind of data sharing system would be really handy - hundreds of autistic patients must have had structural MRIs done over the past few years for research purposes, and if they were publicly available (anonymously of course) it would make it easy to extend this.
I didn't read the paper so forgive my errors but:
1. Did they use cross validation to determine accuracy? It appears they used the training error rate. Error rate on training data is usually a VERY BAD estimator for error rate on future, unseen data.
2. Distance from the separating hyperplane is a very subtle notion to interpret. Caution! A related technique, "Relevance Vector Machine" (RVM), allows one to interpret distances as true probabilities.
3. Here is a link to Bayesian approaches, including RVMs, and associated MATLAB algorithms for this type of machine learning: http://www.gaussianprocess.org/gpml/
4. Back to point #1. Black box machine learning techniques must be used with extreme caution, although they are wonderful tools.
antianticamper: They say that they used two cross-validation approaches:
"The performance of the classifier was validated using the commonly used leave-two-out cross-validation approach. This validation approach provides robust parameter estimates particularly for smaller samples. In each trial observations from all but one subject from each group were used to train the classifier. Subsequently, the class assignment of the test subjects was calculated during the test phase. This procedure was repeated S = 20 times (where S is the number of subjects per group), each time leaving observations from a different subject from each group out. The accuracy of the classifier was measured by the proportion of observations that were correctly classified into patient or control group...
Classifier performance was evaluated using basic receiver operating characteristics (ROC) graphs as well as permutation testing. Permutation testing can be used to evaluate the probability of getting specificity and sensitivity values higher than the ones obtained during the cross-validation procedure by chance. We permuted the labels 1000 times without replacement, each time randomly assigning patient and control labels to each image and repeated the cross-validation procedure. We then counted the number of times the specificity and sensitivity for the permuted labels were higher than the ones obtained for the real labels. Dividing this number by 1000 we derived a p value for the classification."
It is not entirely clear (to me) which method they used to validate each result, however.
So how would the brain of a bipolar woman diagnosed as an adult (by neuropsychiatrists) with both ADD and borderline Aspergers look? Completely different or completely inconclusive? Sure, ADD and ASD seem more prevalent in males, but the study design seems incomplete to me. Inquiring minds need to know.
WV: diends - what comes after they show dimovies
Hi Neuroskpetic -
Nice posting.
If I understand correctly, this study had no participants with functional mental retardation, which comprises between 20 - 40% of children with autism. Do you have any thoughts as to how / if this might affect the application of the scan across the spectrum?
I was also curious if you could go a bit more into detail about the actual differences found; i.e., was this primarily a finding involving white / grey matter where it 'shouldn't' be?
Thanks a lot.
- pD
This article has attracted alot of negative attention when we should be saluting the authors for thinking in a novel manner and hoping to help families of individuals with Autism. There are a couple of points that I wanted to highlight.
As another blogger pointed out - the article by Carl Heneghan, although interesting and statistically correct, does not apply here. There is no mention in the article nor anywhere in the press that the authors suggest this potential test would be used as a screen in the whole population - so therefore Carl's point is invalid. From the media coverage I have seen (and I have been following it closely) the authors have never said this potential technique would replace the standard approaches currently used - however it might help with the process - anyone who works in the diagnosis of adults with ASD they know how difficult this is. They will also know how important a diagnosis can be to people.
This and the previous study by the same author are the first of their kind and therefore LOTS more work in this field obviously needs to be conducted before it's 'rolled out' in the NHS - however, it does allow us to think that there might be additional methods available when diagnosing an individual with ASD. Current methods relay on psychiatric interviews, the ADI-R and the ADOS-G. A point that alot of people forget is - how reliable are clinicians with their diagnosis of ASD - do all clinicians who use the ADI-R and ADOS-G maintain the high reliability suggested by Lord.
It is interesting to think that all of us researchers, when applying for our grants, talk about how one day we want to be able to 'translate' our findings from the lab to the clinic - however when a group actually report on a finding that could one day have this potential - the scientific/psychiatric community start to criticize.
Post a Comment