The Judenzählung was a census of the German military carried out in 1916, at the height of the First World War. It was designed to measure the number of Jewish soldiers in the army.The background to this was the feeling, very powerful in Germany at that time, that the Jewish German minority were "unpatriotic" or "traitorous", and were dodging military service or avoiding front line combat.
The survey was completed, but the results were not published, apparently because they revealed that, contrary to popular belief, Jews were at least as likely as non-Jews to be serving in the army, and were overrepresented on the front lines as well. (Some of the data did emerge after the war, however, when they were criticized for being inaccurate by Jewish groups. 12,000 German Jews died in battle during the War.)
This is an exceptional example of publication bias, but in essence it's no different to what happens when academic or corporate researchers decide not to reveal data which they're not happy with, for whatever reason. The best-known culprits are pharmaceutical companies who often decline to publish data showing that their drugs don't work, but it's a problem that affects most of science, and Big Pharma are certainly not the only ones doing it.
One solution is to have scientific journals or websites dedicated to publishing "negative" results, and there are several, but this still relies on people choosing to reveal their data. It seems to me that, ultimately, the best way to combat publication bias is to require the pre-registration of studies, so that everyone knows in advance what research is being done, and "missing" results can be noticed.
[BPSDB]
10 comments:
Alternatively, you can copy the Global Warming hysterics, and not only suppress your opponents' research, but fake your own.
When their private e-mails got leaked there was no evidence of them doing that, but I suppose it's possible they have some kind of secret code where innocent words have sinister meanings. Maybe "hello" stands for "good luck suppressing those studies" and "cheers" = "I'm off to falsify my data"?
This is a huge problem in all of empirical science. I think journals like "Journal of Articles in Support of the Null Hypothesis" are a stupid idea. Negative results in neuroscience should be published in neuroscience journals, not in a "null results" journal, as if the paper stops being interesting to its natural readership because it failed to reject a null hypothesis.
@ Yigal; I agree, but this was addressed by the alternate suggestion that studies could be preregistered. Of course this seems to require knowing where you want to submit the article prior to running the experiment, submitting a proposal and then (finally) running it. It may have misinterpreted something here, but it seems like an idealistic (read unrealistic?) situation since very few people know in advance where they want to submit (indeed where they submit depends largely on the outcome, thus compounding the issue!)
This likely isn't going to go away, but if major journals required a certain number of "proof of null", experiments, this could help diminish the file drawer problem.
Neuroskeptic,
Just curious why skepticism stops at neuro- and pharma- but doesn't extend to enviro-? From what I've read there is plenty of Bias 2009 Style in these emails.
From WSJ:
...the European Commission's most recent appropriation for climate research comes to nearly $3 billion, and that's not counting funds from the EU's member governments. In the U.S., the House intends to spend $1.3 billion on NASA's climate efforts, $400 million on NOAA's, and another $300 million for the National Science Foundation. ... California—apparently not feeling bankrupt enough—devot[ed] $600 million to their own climate initiative...
Also from WSJ:
In the 1990s, CRU (University of East Anglia's Climate Research Unit) director Phil Jones helped bring in £1.9 million ($3.1 million) for climate research. But in this decade, according to one of the leaked documents, the total shot up to £11.8 million, including grants from the U.K. National Environmental Research Council, the U.S. Department of Energy and NATO. Another leaked spreadsheet for CRU researcher Tim Osborn shows a similar pattern. Between 1994 and 2000, Mr. Osborn secured research contracts totaling £173,881. Between 2001 and 2007, the last year covered by the file, his haul jumped to £764,055.
This is not "innocent" - it's BIASed!
Vested Interest, not sex, is true primal sin. V.I. is the mother of of all biases. US government pursues its goals through "allegedly unbiased" NIMH research. Holistic and "natural" remedies are biases indistinguishable from lies. No one talks about monumental psychotherapeutic research bias.
Clearly, BigPharma doesn't have monopoly on bias.
Why then BigPharma's bias is treated harsher than everyone's? Is it because crucifixion of Bigpharma tangentially absolves others from their sins?
John: I don't think preregistration requires authors to choose the journal ahead of doing the study. Many clinical trials are now being pre-registered (http://clinicaltrials.gov/) and most of the top medical journals have a policy of refusing to publish any trial that wasn't preregistered. But the trials weren't registered to any one journal.
Mandatory preregistration of non-clinical research is not on the horizon yet, but it's what I advocate, and I think it could happen. It wouldn't need to be every journal. Even if it was just Nature and Science, for example, that required pre-registration, it would make a huge difference.
ML: My scepticism stops at neuroscience & related stuff because that's what I know about. I'm not a climate scientist so I'm not able to offer informed comment on climate... as for the emails, I've yet to see anything which suggests the scientists in question were doing anything outside the academic norm & there's certainly no evidence they did anything which means their conclusions are cast into doubt. I'm not sure why the leaked "revelation" that Phil Jones and Mike Osborn got a lot of grant money is evidence of bias.
As for Big Pharma's bias being treated worse than other people's, it shouldn't be, although the sheer scale of it is often rather attention-grabbing.
As with everything else, by itself the "revelation" that Phil Jones and Mike Osborn got a lot of grant" is not evidence of bias. In combination with leaked e-mails and other researcher's data received $ are ground for skepticism. It's 13th chime principle that puts in doubt previous 12.
problem is that very well designed studies can go absolutely troppo in the discussion, there's a 'experiment' that did just that with what was obviously the priming effect, but the researcher claimed it was 'nature's innate influence on mankind'. Sure, that could be combated by having somebody else write the discussion, but even then, they could just screw up the actual experiment.
Post a Comment