Tuesday, 31 May 2011

Vaccines Cause Autism, Until You Look At The Data

According to a much-discussed new paper, vaccines may cause autism after all: A Positive Association found between Autism Prevalence and Childhood Vaccination uptake across the U.S. Population.

The author is Gayle DeLong, who "teaches international finance at Baruch College, City University of New York", according to her profile as a board member of anti-vaccine group SafeMinds. She correlated rates of coverage of the government recommended full set of vaccines in the 51 US states including Washington D.C., with registered rates of autism in those states six years later.

Uh-oh - there was a correlation between vaccination in two year kids, and the rate of autism in the state six years later, when those kids were eight. As the abstract says:
The higher the proportion of children receiving recommended vaccinations, the higher was the prevalence of AUT... The results suggest that although mercury has been removed from many vaccines, other culprits may link vaccines to autism. Further study into the relationship between vaccines and autism is warranted.
Sounds rather scary. Until you look at the data, helpfully provided in the paper. First up, here's the scatterplot of all of the vaccination rates and all of the autism-six-years-later rates:

There's more than 51 data points as you can see: there's actually 355 because each state had seven different datapoints (1995 vaccines vs 2001 autism though to 2001 vs 2007). This scatterplot shows no correlation. You can tell just from looking at it, but the correlation coefficient confirms this, as it's a tiny r 0.012 (from a possible range of 0 to 1).

To be fair, that's a very noisy measure, because each state has unique characteristics, so the effect of vaccines will be diluted. However, it's still a useful sanity check, and shows that there can't be a major effect, otherwise it would be too big to get diluted.

To get around this I next looked at the change in the rates of vaccination from one year to the next, and correlated that with the corresponding change in future rates of autism, within each state. A "change" of 1 means no change, 0.5 means it halved and 2 means it doubled, etc.

Zilch. Correlation coeffiencent r is 0.034.

Maybe the changes year-to-year were too small? So I checked the changes between the last year, and the first year.

This made the changes bigger, because more tends to change over six years than in just one. And, to be fair, this does produces a slightly stronger vaccine-autism effect... but it's still tiny. The correlation coefficient here, r, is 0.18 which means that vaccination changes accounts for 3% of the variability in autism changes (r^2 = 0.034.) The p value is 0.20, statistically insignificant.

My conclusion is that this dataset shows no evidence of any association. The author nonetheless found one. How? By doing some statistical wizardry.
The statistical model used took into consideration the unique characteristics of each state. For example, each state had a unique mixture of pollution, which may have affected the prevalence of autism, yet such an effect was not included in this study. A fixed-effects, within-group panel regression (Hall and Cummins 2005) controlled for these unique yet undefined characteristics by deriving a different starting point (intercept) for each state.

The 51 different intercepts - one for each state - reflected the base level of autism or speech disorders occurring in that state that were not explained by the other independent variables (vaccination rates, income, or ethnicity). The model then produced a single relationship between the independent variables and the prevalence of autism or speech disorders.
OK, that's all very fancy, but when the raw data shows zilch and you can only find a signal by "controlling for" stuff, alarm bells start ringing. Given sufficient statistical analysis you can make any data say anything you want.

If the author had given details of the methods, and explained why she chose to control for the variables she did, and not others, that might be different. But she didn't. Nor did she justify only looking at the effects six years later, when five or seven or ten would be just as sensible... and so on.

(Note: whenever I've said "autism", that's my shorthand for autism + SLI, which is what the paper looked at; autism alone data are not presented. Note also that by "vaccination %" I mean "% who got the full vaccine schedule"; the other kids may have got vaccines, just not all of them.)

ResearchBlogging.orgDelong G (2011). A Positive Association found between Autism Prevalence and Childhood Vaccination uptake across the U.S. Population. Journal of toxicology and environmental health. Part A, 74 (14), 903-16 PMID: 21623535

32 comments:

Anonymous said...

So uh, is this journal peer reviewed? Why on earth would a journal on toxicology publish an article written by a second rate economist, much less one with such shoddy statistical work?

Doesn't do much to help my feeling that you can get literally anything published as long as you have a minimum of contacts and have positive results for any hypothesis.

Catherina said...

thankyouthankyouthankyou - I will shamelessly link to this as soon as I find the time to post. I think the "peers" indeed would deserve some looking into. It appears anti-vaccine minded scientists are also to be found on editorial boards (or maybe they just don't care and an anti-vaccine review issue will raise impact factor) these days. Frustrating!

Anonymous said...

I'm a PhD statistician and I have to admit that I found the statistical analysis in this paper to be quite reasonable.
If I were to prepare my own analysis of this data I would also be sure to allow for a distinct intercept for each state in the model, since that allows you to investigate whether a change in the vaccination rate within a particular state had an effect on the autism rates. Doing this removes some of the noise that may result from other confounding variables at the state level.
I was disappointed to see you present all of the data points in a single scatterplot because that actually obscures much of the information contained in the data. Your further "examination" of this combined data where you claim to be looking really hard for correlation is clearly a poor attempt to dazzle us readers into thinking that you've done something sophisticated when in fact you have not. I am really disappointed in the quality of this post.

Neuroskeptic said...

Anonymous: I just re-ran the analysis using a linear regression with each state as a dummy variable 0 or 1 and each year "band" as a dummy variable 0 or 1.

This replicates the paper's headline finding: vaccination predicts autism with a coefficient of 0.017, p value less than 0.001.

However. This model is overfitted to the data: it also "finds" that every single state and year band predicts autism, with much larger coefficients and in almost all cases the same or lower p values as the autism one. Vaccines is the worst predictor in the model.

Clearly this model is wrong: the state you live in can't, in itself, affect autism rates; it must be a proxy for other factors. If you take this model seriously everyone ought to move to New Hampshire because the model says that New Hampshire halves the rate of autism with a p of less than 0.001! No-one takes that seriously, so why would we take the worst predictor (vaccines) seriously?

Neuroskeptic said...

And, if you weight the data by the 2010 population of the state, which arguably you ought to do, because small states are noisier, the effect gets a lot smaller: the p value drops to 0.079.

Admittedly, I'm now kind of doing what I always warn against, running lots of different analyses post-hoc.

quicksilver said...

Easy to rubbish people who look for causes.

The fall in health if USA children is astonishing.

So what is the cause or are we happy not to know?

The unrelated deaths in Germany and Europe indicate something is going wrong with our health policies today.

When I had vaccines many years ago one only worked first time with absolutely no one asking questions on safety or efficacy.

Today many countries halt temporarily vaccines as babies die and we need up to one shot a year for each vaccine to keep illnesses away.

Jon Brock said...

Surprised you didn't mention the "ecological fallacy".

"The term comes from a 1950 paper by William S. Robinson.[9] For each of the 48 states + District of Columbia in the US as of the 1930 census, he computed the literacy rate and the proportion of the population born outside the US. He showed that these two figures were associated with a positive correlation of 0.53 — in other words, the greater the proportion of immigrants in a state, the higher its average literacy. However, when individuals are considered, the correlation was −0.11 — immigrants were on average less literate than native citizens. Robinson showed that the positive correlation at the level of state populations was because immigrants tended to settle in states where the native population was more literate. He cautioned against deducing conclusions about individuals on the basis of population-level, or "ecological" data."

http://en.wikipedia.org/wiki/Ecological_fallacy

Jake said...

Trying scrutinize the data from the table given by the authors seems a little pointless because the table only includes a few of the most relevant variables, not all that the authors actually had access to or used in their analysis. It's perhaps worth mentioning that if you fit a multilevel model to the table data (which, as anon mentions, is definitely how this data ought to be analyzed), the effect absolutely is not there when you simply control for time (t = -0.195). Of course, this is only the starting point for the authors, who go on to control for all kinds of other variables that we simply don't have access to.

The main thing I'm a little puzzled by is why they accounted for time in this way. You mentioned in your post that it was fishy that they inexplicably chose a time lag of 6 years, and I guess I agree, but more importantly, why choose a single time lag at all? If they had access to the full time series data for both continuous variables, wouldn't it make the most sense to analyze that data? Granted, time series analysis is far from my specialty, but this strikes me as a limiting way to look at the data.

praisegod barebones said...

@Anonymous (first comment): it's not Brad deLong.

mariawolters said...

Neuroskeptic, I haven't had time to look at the paper, but this looks like a classical multilevel model to me, and I would agree with Anonymous - it's an appropriate way of analysing surveys. The linear regression you report is related to multilevel modelling, but it's by no means a replication.

If you really want to replicate the statistical analysis, you should use a proper multilevel modelling tool, which is accessible for free in R, package lme4. R also has tools for producing scatterplots by state.

There are many ways in which one could misinterpret the findings of a multilevel model, and if you do want to write a reasonable statistical critique, you should look at those. For example, were there any outliers in the data? How frequent was autism? Were any states underrepresented? Should partial pooling have been done?

If you'd like to get in touch to discuss further, let me know - my email address is firstname . lastname at gmail dot com

Anonymous said...

well... I can't even read this paper. I wonder if authors would be kind to make the data publicly available... not to mention the actual modeling/analysis script (and its version ;-) )

Anonymous said...

I would be interested to know how anyone thinks injecting infants on purpose and making them contagious to others for up to six weeks (in the case of the Varicella vaccine) is "preventing" the spread or "protecting" them from infectious disease? Your photo is based on fear mongering.
Also, you may want to inform Sanofi Pasteur Inc, makers of the Tripedia vaccine who have AUTISM listed as a potential adverse reaction on page 11 of their label. What other "evidence" is needed when the makers themselves acknowledge a link?

Liz Ditz said...

Some other problems with the DeLong paper, which Neuroskeptic (NS) wouldn't have known, being a non-USian.

NS wrote, registered rates of autism.

Well, no. The US doesn't have a national registry of autism cases. Why not? The US does not have national health care.

Here's the quote from DeLong's method section:

Prevalence of autism To determine autism prevalence by U.S. state, the number of 8-year-old students classified with either (1) autism or (2) speech or language impairments (speech disorders) was divided by the total number of 8-year-olds in the state. The number of children with disabilities came from the U.S. Department of Education, Office of Special Education Programs (2007) and the total number of students came from the U.S. Department of Education, National Center for Education Statistics. Although the diagnosis of autism is usually made when a child is 3 or 4 years old, some children are not diagnosed until they are older. Children who receive a diagnosis of autism usually do so by the time they are 8 years old. The category of speech or language impairments was included with autism, because these impairments are closely linked to autism (Conti-Ramsden et al. 2006; De Fosse et al. 2004; Herbert et al. 2007)

Translation:

DeLong used reporting statistics required by the US federal legislation, Individuals with Disabilities Education Act (IDEA). Under that act, each school must provide data on children who have an Individual Education Plan (IEP), including the students' "primary classification". The data are aggregated at a national level.

There are things to note here:

1. Children whose primary classification under IDEA as having autism may or may not, clinically speaking, have autism.

2. Some children whose primary IDEA classification is autism may also have language difficulties and receive speech therapy, which is entirely different than having a primary IDEA classification of "speech or language impairment" (SLI).

3. A child with a primary IDEA classification of "speech or language impairment"(SLI) is most often a child with difficulty in fluency (that is, stuttering or stammering), articulation (severe lisp or significant oral dyspraxia) or voice (speech production) not autism .

4. The deDe Fosse et al. paper deLong cites to justify lumping SLI with autism does not support her argument. It deals with Specific Language Impairment not the broad IDEA category of speech/language impairment.

5. Thus, there's no reason to combine SLI with autism...unless you need to make the numbers work out.


DeLong is lumping apples and tomatoes together, and claiming it's a valid combination, because they are both pommes.

Liz Ditz said...

I really, really wonder if deLong added the SLI category to make her numbers work out.

I happened to be looking at California special education enrollment for another project. Website http://dq.cde.ca.gov/dataquest/dataquest.asp

For school year 2009-2010:

All students with autism, ages 0-22 (when you age out of coverage) = 0.07% of total enrollment.

All students with Speech or Language Impairment, ages 0-22 = 2.71% of total enrollment.

Here are the definitions:

Autism (AUT): Autism means a developmental disability significantly affecting verbal and non-verbal communication and social interaction, generally evident before age three, that adversely affects educational performance.

Characteristics of autism include -- irregularities and impairments in communication, engagement in repetitive activities and stereotyped movements, resistance to environmental change or change in daily routines, and unusual responses to sensory experiences. The term does not include children with characteristics of the disability serious emotional disturbance (SED).

If a child manifests characteristics of the disability category "autism" after age three, that child still could be diagnosed as having "autism" if the criteria in the above paragraph are satisfied. (34 CFR Part 300.5).



Speech or Language Impairment (SLI): Speech and Language Impairment means a communication disorder such as stuttering, impaired articulation, language impairment, or a voice impairment, which adversely affects a child's educational performance. (34 CFR Part 300.5).

deevybee said...

Since children with SLI typically show evidence of slow language development before they are vaccinated, it does seem odd to include them.

deevybee said...

A further thought: epidemiological studies show SLI is actually more common than autism; did the author give breakdown of numbers with each diagnosis? Does seem that this study is claiming vaccination can affect a child's development a year or so prior to the vaccination.

Neuroskeptic said...

Thanks for all of the comments. It looks like I dropped the ball on this one and focussed on the statistics, which were OK, over the actual design, which was dubious. Lumping SLI with autism is really odd, when you think about it.

jre said...

It's true, of course, that good study design is fundamental, and requires some understanding of the problem. Correct execution of the statistics, in contrast, simply requires technical proficiency. So it is with some hesitation that I ask: could you point to the data in some convenient format, such as an R dataframe? I found a pdf of the paper, but I'm too lazy to reformat the data.

Also -- I am encouraged to see that the comments have focused mostly on the substance of the study and not on the author's background. Not because I think it's irrelevant -- association with SafeMinds is a major red flag -- but because that discussion quickly goes down a rabbit hole, never to return.

Neuroskeptic said...

I've got the data in Excel and SPSS. If you email me I'll send you them tomorrow when I get access to them.

RINKEVICHJM said...

Is there a control for comparing use of acetaminophen versus aspirin or ibuprofen? These pain relievers are ordinarily recommended for use to reduce pain and inflammation produced by vaccinations. And acetaminophen has been indicated by others as a probable cause for autism.

jre said...

Thanks! I just have this compulsion to fiddle with the stats, even when we all know that great stats won't save a flawed design.

Of note, Orac over at Respectful Insolence has directed his blinky lights to this study.

Matt Carey said...

If this went to referees, they clearly failed as this paper is so clearly bad.

There are many points--even beyond those you make--which call this study into question. But, take everything at face value. The simple test is to just look at the data in table 1.

http://leftbrainrightbrain.co.uk/wp-content/uploads/2011/06/Table1a1.png

The "vaccination rate" (they have a very odd definition) goes up dramatically in the first two years of the study. The "autism+SLI rate" does not.

If there were a real association between the two, it would be clear in those first columns of data.

Neuroskeptic said...

That's what I was originally getting at, with my scatterplots: there's no correlation between change in vaccination and change in autism.

Matt Carey said...

"Lumping SLI with autism is really odd, when you think about it."

Very true. Consider also their definition of "vaccination rate". It is defined as those who got the full 1995 schedule of vaccines vs. those who missed 1 or more vaccines.

By this definition the "vaccination rate" climbs dramatically in the first two years. This is because they are working with a new schedule and it takes time for states and pediatricians to adopt it.

The funny thing, the autism/SLI rates don't go up in the first two years, the exact time that the "vaccination rate" is going up so fast.

The time trends show basically two time constants--a fast time constant for the uptake of the new schedule and a slow time constant for the response of the autism/SLI rate. That alone should have tipped off the referees that the conclusions here were wrong.

Matt Carey said...

"That's what I was originally getting at, with my scatterplots: there's no correlation between change in vaccination and change in autism."

Absolutely.

One might counter argue that the association is more subtle and would only appear after controlling for factors like income and ethnicity.

That's why I looked at the first two years alone. Any association would be clear, even in the raw data, given the jumps in vaccination rates.

By the way, consider their choice of data sources for ethnicity. The author ignored the U.S. census (which the author used for other data) and instead used the data from the immunization survey.

Why do that? Was the author sloppy, or did this help to obtain the "result".

Note that often surveys of this sort will oversample minorities. It isn't clear whether this is true in the immunization survey (it must be spelled out somewhere, but I haven't seen it). If so, did the author use the oversampled data or the raw data for ethnicity?

Liz Ditz said...

Also see yesterday's (June 7 2011) post from Emily J. Willingham, PhD at The Biology Files: http://biologyfiles.fieldofscience.com/2011/06/gayle-delong-safeminds-board-member-and.html:

The conclusion of this paper also should have received more reviewer notice, as it is wall-to-wall careless and unsupported speculation, in many cases about ideas that have been debunked

Liz Ditz said...

And today's post from Orac at Respectful Insolence, that http://scienceblogs.com/insolence/2011/06/more_bad_science_in_the_service_of_the_discredited_idea.php. Good commentary from among others, Prometheus.

Chris said...

The lumping of kids with Specific Language Impairments into autism is mind boggling. Hey, I got speech and language services in 2nd and/or 3rd grade in California back in the early 1960s. For me it was because of a delay due to a temporary hearing loss (ear infection not picked up while moving across the country and my dad getting ready to go to Vietnam, somehow me being very quiet was thought to be helpful at a time of stress).

Now to ask some questions of those who did not bother reading the article:

quicksilver: Today many countries halt temporarily vaccines as babies die and we need up to one shot a year for each vaccine to keep illnesses away.

Cite please on the countries that temporarily stop vaccines. I know that it happened in Japan because they blamed two SIDS deaths on the DTP vaccine. That experiment was halted when 41 babied died of pertussis. See PMID 15889991 .

Anonymous: I would be interested to know how anyone thinks injecting infants on purpose and making them contagious to others for up to six weeks (in the case of the Varicella vaccine) is "preventing" the spread or "protecting" them from infectious disease?

Please provide the PubMed Identification number of the paper that shows this happens. And also what it has to do with the paper under discussion.

RINKEVICHJM, that would make the date on this paper even more meaningless than it already is. Kine of trying to clear up a mirror by throwing mud at it!

Anonymous said...

http://adf.ly/28LAD autism

Shine said...

I'm a speech therapist and have quite a few students with autism spectrum disorder on my public school caseload. The immunization issue has been looked at for years and that strong correlational link has not been found. Have there been significant correlational studies done with the autism rate and the usage of prenatal ultrasound (sometimes multiple times, starting at just a few weeks gestational age?) Ultrasound has been used for decades, but has continued to increase and increase in how often and how early it is used. Now, there are 3-D ultrasound photos taken in some areas for no medical reason, just prenatal photos for an album. Immigrant families moving to this country (such as Somali refugees in the Minneapolis area) seem to have a major surge in their autism rate when they move here.....which could be accounted for by the practices of modern prenatal care (which they would not have experiences in their country of origin). Has a possible link with ultrasound been looked at? This tragic continued increase in the number of children with autism is gutwrenching. My heart goes out to any family facing this issue (we have a nephew with Asperger's).

karyn said...

I really don't know much about this subject but I came across your discussion after I chose this article for my final philosophy paper. I just wanted to share that last year my twelve year old son vomited for nineteen days, starting twenty-four hours after receiving vaccines and had to be hospitilized twice and given IV fluids. He had no fever and no other signs of illness and his doctor believed it was a reaction to the vaccines. He received botox injections in his legs a year later for muscle spasticity and vomited for fourteen days. Maybe vaccines don't cause problems in all children but maybe some kids just can't tolerate them the way others can. It doesn't have to be a black/white situation. Penicillin (sp?) was life saving at one time but others were deathly allergic. Maybe everybody is right.

karyn said...

I really don't know much about this subject but I came across your discussion after I chose this article for my final philosophy paper. I just wanted to share that last year my twelve year old son vomited for nineteen days, starting twenty-four hours after receiving vaccines and had to be hospitilized twice and given IV fluids. He had no fever and no other signs of illness and his doctor believed it was a reaction to the vaccines. He received botox injections in his legs a year later for muscle spasticity and vomited for fourteen days. Maybe vaccines don't cause problems in all children but maybe some kids just can't tolerate them the way others can. It doesn't have to be a black/white situation. Penicillin (sp?) was life saving at one time but others were deathly allergic. Maybe everybody is right.