That's publication bias. A simple but powerful remedy is to require everyone to publically announce their trials before the data comes in. The USA has led the way in this, with the public clinicaltrials.gov database, and for several years it's been a legal requirement that all clinical trials conducted in the USA must be pre-registered there, and that the results have to be uploaded when they arrive.
A new study by Bourgeois et al used this database to assess the scale of non-publication: Outcome Reporting Among Drug Trials Registered in ClinicalTrials.gov. Out of over 500 clinicaltrials.gov trials they looked at, 66% of them ended up getting published eventually. (The trials all ended before 2006, so they've had 5 years to get published and if they haven't by now, it's unlikely they ever will.) Is that a lot? Well, it's better than I'd expected, but it's still 33% too low.The odds of getting published varied depending upon the type of drug, though. Trials of proton-pump inhibitors and cholesterol lowering drugs had the best chances. Antidepressants were a bit less publishable; antipsychotics were markedly worse; and only just over half of the vasodilator trials did.
This is interesting data, and it should remind us that publication bias, although often discussed nowadays as a problem with trials of antidepressants, is by no means limited to those drugs and in fact antidepressant trials (at least the ones starting after 2000 and completed by 2006) are fairly middle-of-the-road in terms of % publication rates.
Publications resulting from drug company-funded trials were also more likely to be positive (85%) than were trials bankrolled by the government (50%) or non-profit organizations without Pharma "contributions" (61%). This doesn't prove that drug companies are biasing publication - maybe they just really do get more positive results - but, well, it's not exactly reassuring.
Why is non-publication still a problem, given that people are required by law to release their trial protocols and results on clinicaltrials.gov? The problem is that clinicaltrials.gov doesn't appear on PubMed, and medical science works on the rule of "PubMed or it didn't happen". Someone searching for papers about "drug X for disease Y" - which I suspect accounts for the vast majority of clinical paper downloads - will still only get told about the trials that the authors chose to publish.
Is there an answer? We could in theory force people to write their results up and submit them to a journal, and we force journals to publish them, but that would be unworkable and incredibly unpopular. But why not simply publish the results from clinicaltrials.gov?
Whenever someone uploads their results to the database (as they legally must), clinicaltrials.gov could automatically use them to generate a mini-paper and publish it online. There would be a few tricky issues to sort out - you'd have to be careful that it didn't lead to the same results getting published in multiple places, for one - but so long as these reports were indexed on PubMed, it would solve the fundamental problem.
Bourgeois FT, Murthy S, & Mandl KD (2010). Outcome Reporting Among Drug Trials Registered in ClinicalTrials.gov. Annals of internal medicine, 153 (3), 158-66 PMID: 20679560
11 comments:
This is a wonderful idea which would improve science and health no end.
Thats why it won't happen. \snark.
Well, mandatory pre-registration was a wonderful idea 10 years ago and now it's a reality - in Europe as well as the US (but the European database isn't publicly viewable at present.)
Doesn't look like anyone takes the posting of results requirement seriously on clinicaltrials.gov. Just did a search by condition - depression. 1214 closed (most are completed) studies, 30 with results.
Do FDA and EMEA consider all studies during approval process?
Pharmaceutical companies are obliged to provide all their studies to the FDA (not sure about the European regulators) which is a good thing, but those studies are not necessarily in a usable format for analysis and they aren't made publically available.
Once a drug is approved and you want to look at, say, cost effectiveness like NICE does then there is no obligation to provide the data.
Registration if trials is a start but there is increasing movement out of the US and EU to do trials abroad. We need to legally oblige the pharmaceutical companies to not only provide all relevant trials performed before licencing but also that those trials should have been centrally registered before being performed whatever country they were performed in.
Again that is still only a start - we need all clinical trials to be registered whether performed by pharmacuetical companies or academics - and we need all that data to be available, even studies performed after licencing, to fully assess the evidence.
For a system like the UK's NHS which needs cost effectiveness data to make recommendations we're still a long way off being able to actually make fully informed rational decisions - which is just how the pharmaceutical companies want it, and how their lickspittle politicians will endeavour to keep it.
one could assume the authors/pharma companies *chose* not to publish, but isn't it also the case that conventional journals are rarely interested in negative/null results?
anonymous: Big-name journals certainly tend to go for positive data but there are a lot of journals nowadays (including journals specifically dedicated to negative results etc.) and it's usually possible to get stuff published somewhere.
However it's true that there are some journals which are so bad that people would actually rather not have a publication in them - they actively make your CV worse.
pj: I believe that all trials involving EU patients have to be registered with the EU authorities & the results sent in. The data is available to certain national "Competent Authorities" but I'm not sure if NICE is one of them.
However you're right that "trial tourism" is a growing problem. I suppose what the US/EU authorities can do to stop that is to say "We will not license any drug for which there exist any unregistered trials, wherever they take place". Although this wouldn't help with post-marketing trials.
There's still the well written paper in PLOS, Why Most Published Research Findings Are False
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/
which to my mind is the gospel of how to read studies.
There is also a publication bias in psychotherapy research, especially for CBT publications on adult depression:
http://bjp.rcpsych.org/cgi/content/abstract/196/3/173?maxtoshow=&hits=10&RESULTFORMAT=&fulltext=cuijpers&searchid=1&FIRSTINDEX=0&resourcetype=HWCIT
...
Method: We examined effect sizes of 117 trials with 175 comparisons between psychotherapy and control conditions....
Results: The mean effect size was 0.67, which was reduced after adjustment for publication bias to 0.42 (51 imputed studies). Both Begg & Mazumbar’s test and Egger’s test were highly significant (P<0.001).
Conclusions: The effects of psychotherapy for adult depression seem to be overestimated considerably because of publication bias.
Pharmacology as a field needs to reevaluate many of its current practices. And stricter oversight is indeed necessary.
There are many studies out there which seem to conflict with, and sometimes even completely contradict what are currently standard practices in medicine and psychiatry. Here's one interesting study:
http://www.ncbi.nlm.nih.gov/pubmed/17502806
This is only one study, so one couldn't draw many conclusions: perhaps antipsychotics are only efficient in the short-term treatment of psychotic episodes?
You would think there would be more than just one article citing this interesting study.
I think one of the problems is that a pharmaceutical company does not experience the same external pressures that other industries experience when it comes to always having to be ahead of the game and to change and adapt as quickly as the evidence rushes in.
If IBM becomes lazy with its research and comes out with a processor that doesn't quite work, they will lose clients fast.
The effect is not so immediate for a pharmaceutical company. In the long run, the better treatments stay. But in the short term, drugs are endlessly being released (and advertised) to the general public that could have used more time with test subjects. Or maybe not have been released at all: drugs that only seem to work for some people, drugs that do only marginally better than placebos, and in the worst cases, drugs that are found to cause severe health problems after long-term use.
An inevitable outcome of this—aside from overmedication—is that many people become so untrusting of pharmaceutical companies that they turn instead to completely unfounded and possibly dangerous forms of treatment.
M.P. : Right, publication bias is a problem everywhere. I think therapy trials should be subject to exactly the same regulations about registration etc. as drug trials.
Noam: Thanks for the link to that very interesting study, I hadn't heard of it... oh wait, the MacGuffin wrote about it earlier this year.
Post a Comment