How Not To Do Great Science (The Lost Post)

This post was originally published on Discover Magazine on September 16th 2013, but has since vanished (although most of my other Discover posts are still available). Luckily, I saved a backup. So here's the original "How Not To Do Great Science".

---

This post is a bit special. For the first time ever, I've collaborated with an artist, Erene Stergiopoulos. Her webcomic is here and she's on Twitter here. I think you'll agree that the artistic standard is a little higher than I usually achieve. Anyway, here's what we did:

It would be silly to expect that every architect should finish buildings at a certain rate. That would make it impossible to anyone to build certain things. Some things take longer to build than others, and most great things take a great deal of time. Faced with a sufficiently demanding quota, builders might be reduced to rushing out follies that might look impressive from a distance, but that are no more than hollow shells. Yet, as silly it would be to make uniform demands of architects, this is what is happening to scientists.

Rather than build, scientists are expected to publish - and publish fast - or perish. My worry (and that of many others) is that the pressure to publish often fundamentally changes not just how much scientists write, but what they can write about. It turns researchers into prolific doers of small deeds, but it leaves them little time to think about, let alone complete, great works. Though the mills of God grind slowly...

Yet the problem is not just the speed of science today, but also the direction: go to a scientific conference and you'll see perfectly good data in the process of being oversold, misinterpreted, and p-hacked into a 'publishable' form.

Much has been said about how this leads to false positives - impressive follies that don't stand up to scrutiny. What's less discussed - and the point of this piece - is the opportunity cost. New theories come out of attempts to explain 'negative' data - negative from the perspective of the old theory. Null results are the foundations of future progress, but only if they are allowed to lie there awhile; not if they are torn up and used to prop up tottering old structures.

Unilaterally Raising the Scientific Standard

For years, I and others have been arguing that the current system of publishing science is broken. Publishing and peer-reviewing work only after the study's been conducted and the data analysed allows bad practices - such as selective publication of desirable findings, and running multiple statistical tests to find positive results - to run rampant.

So I was extremely interested when I received an email from Jona Sassenhagen, of the University of Marburg, with subject line: Unilaterally raising the standard.

Sassenhagen explained that he's chose to pre-register a neuroscience study on a public database, the German Clinical Trials Register (DRKS).

His project, Alignment of Late Positive ERP Components to Linguistic Deviations ("P600"), is designed to use EEG to test whether the brain generates a distinct electrical response - the P600 - in response to seeing grammatical errors. The background here is that the P600 certainly exists, but people disagree on whether it's specific to language; Sassenhagen hopes to find out.

By publicly announcing the methods he'll use before collecting any data, Sassenhagen has, in my view, taken a brave and important step towards a better kind of science.

Already, most journals require trials of medical treatments to be publicly pre-registered, and the DRKS is one such registry. This study, however, is 'pure' neuroscience with nothing clinical about it, so it doesn't need to be registered - Sassenhagen just did it voluntarily.

Further, I should point out that he offered to pre-register his data analysis pipeline too by sending it to me. Unfortunately, I didn't reply to the email in time... but that was purely my fault.

I very much hope and expect that others will follow in his footsteps. Unilaterally adopting preregistration is one of the ways that I've argued reform could get started. As I said:

This would, at least at first, place these adopters at an objective disadvantage. However, by voluntarily accepting such a disadvantage, it might be hoped that such actors would gain acclaim as more trustworthy than non-adopters.

Pre-registration puts you at a disadvantage - insofar as it limits your ability to use bad practice to fish for positive results. It means you can't cheat, essentially, which is a handicap if everyone else can.

I don't know if this is the first time anyone's opted in to registering a pure neuroscience study, but it's certainly the first case I know of it being done for an entirely new experiment.

There have, however, recently been many pre-registered attempts to replicate previously published results e.g. the Reproducibility of Psychological Science; the 'Precognition' Replications; and an upcoming special issue of Frontiers in Cognition.

Replications are good, registered ones doubly so - but they're not enough to fix bad practice on their own. To do that we need to work on the source, original scientific research.

Is This How Memory Works?

We know quite a bit about how long-term memory is formed in the brain - it's all about strengthening of synaptic connections between neurons. But what about remembering something over the course of just a few seconds? Like how you (hopefully) still recall what that last sentence as about?

Short-term memory is formed and lost far too quickly for it to be explained by any (known) kind of synaptic plasticity. So how does it work? British mathematicians Samuel Johnson and colleagues say they have the answer: Robust Short-Term Memory without Synaptic Learning.

They write:

The mechanism, which we call Cluster Reverberation (CR), is very simple. If neurons in a group are more densely connected to each other than to the rest of the network, either because they form a module or because the network is significantly clustered, they will tend to retain the activity of the group: when they are all initially firing, they each continue to receive many action potentials and so go on firing.

The idea is that a neural network will naturally exhibit short-term memory - i.e. a pattern of electrical activity will tend to be maintained over time - so long as neurons are wired up in the form of clusters of cells mostly connected to their neighbours:

The cells within a cluster (or module) are all connected to each other, so once a module becomes active, it will stay active as the cells stimulate each other.

Why, you might ask, are the clusters necessary? Couldn't each individual cell have a memory - a tendency for its activity level to be 'sticky' over time, so that it kept firing even after it had stopped receiving input?

The authors say that even 'sticky' cells couldn't store memory effectively, because we know that the firing pattern of any individual cell is subject to a lot of random variation. If all of the cells were interconnected, this noise would quickly erase the signal. Clustering overcomes this problem.

But how could a neural clustering system develop in the first place? And how would the brain ensure that the clusters were 'useful' groups, rather than just being a bunch of different neurons doing entirely different things? Here's the clever bit:

If an initially homogeneous (i.e., neither modular nor clustered) area of brain tissue were repeatedly stimulated with different patterns... then synaptic plasticity mechanisms might be expected to alter the network structure in such a way that synapses within each of the imposed modules would all tend to become strengthened.

In other words, even if the brain started out life with a random pattern of connections, everyday experience (e.g. sensory input) could create a modular structure of just the right kind to allow short-term memory. Incidentally, such a 'modular' network would also be one of those famous small-world networks.

It strikes me as a very elegant model. But it is just a model, and neuroscience has a lot of those; as always, it awaits experimental proof.

One possible implication of this idea, it seems to me, is that short-term memory ought to be pretty conservative, in the sense that it could only store reactivations of existing neural circuits, rather than entirely new patterns of activity. Might it be possible to test that...?

Johnson S, Marro J, and Torres JJ (2013). Robust Short-Term Memory without Synaptic Learning. PloS ONE, 8 (1) PMID: 23349664

Is Medical Science Really 86% True?

The idea that Most Published Research Findings Are False rocked the world of science when it was proposed in 2005. Since then, however, it's become widely accepted - at least with respect to many kinds of studies in biology, genetics, medicine and psychology.

Now, however, a new analysis from Jager and Leek says things are nowhere near as bad after all: only 14% of the medical literature is wrong, not half of it. Phew!

But is this conclusion... falsely positive?

I'm skeptical of this result for two separate reasons. First off, I have problems with the sample of the literature they used: it seems likely to contain only the 'best' results. This is because the authors:

• only considered the creme-de-la-creme of top-ranked medical journals, which may be more reliable than others.
• only looked at the Abstracts of the papers, which generally contain the best results in the paper.
• only included the just over 5000 statistically significant p-values present in the 75,000 Abstracts published. Those papers that put their p-values up front might be more reliable than those that bury them deep in the Results.

In other words, even if it's true that only 14% of the results in these Abstracts were false, the proportion in the medical literature as a whole might be much higher.

Secondly, I have doubts about the statistics. Jager and Leek estimated the proportion of false positive p values, by assuming that true p-values tend to be low: not just below the arbitrary 0.05 cutoff, but well below it.

It turns out that p-values in these Abstracts strongly cluster around 0, and the conclusion is that most of them are real:

But this depends on the crucial assumption that false-positive p values are different from real ones, and equally likely to be anywhere from 0 to 0.05.

"if we consider only the P-­values that are less than 0.05, the P-­values for false positives must be distributed uniformly between 0 and 0.05."

The statement is true in theory - by definition, p values should behave in that way assuming the null hypothesis is true. In theory.

But... we have no way of knowing if it's true in practice. It might well not be.

For example, authors tend to put their best p-values in the Abstract. If they have several significant findings below 0.05, they'll likely put the lowest one up front. This works for both true and false positives: if you get p=0.01 and p=0.05, you'll probably highlight the 0.01. Therefore, false positive p values in Abstracts might cluster low, just like true positives.

Alternatively, false p's could also cluster the other way, just below 0.05. This is because running lots of independent comparisons is not the only way to generate false positives. You can also take almost-significant p's and fudge them downwards, for example by excluding 'outliers', or running slightly different statistical tests. You won't get p=0.06 down to p=0.001 by doing that, but you can get it down to p=0.04.

In this dataset, there's no evidence that p's just below 0.05 were more common. However, in many other sets of scientific papers, clear evidence of such "p hacking" has been found. That reinforces my suspicion that this is an especially 'good' sample.

Anyway, those are just two examples of why false p's might be unevenly distributed; there are plenty of others: 'there are more bad scientific practices in heaven and earth, Horatio, than are dreamt of in your model...'

In summary, although I think the idea of modelling the distribution of true and false findings, and using these models to estimate the proportions of each in a sample, is promising, I think a lot more work is needed before we can be confident in the results of the approach.

How (Not) To Fix Social Psychology

British psychologist David Shanks has commented on the Diedrik Stapel affair and other recent scandals that have rocked the field of social psychology: Unconscious track to disciplinary train wreck,

Lots of people are chipping in on this debate for the first time at the moment, but peoples' initial reactions often fall prey to misunderstandings that can stand in the way of meaningful reform - misunderstandings that more considered analysis has exposed.

For example, Shanks writes:

[despite claims that] social psychology is no more prone to fraud than any other discipline, but outright fraud is not the major problem: the biggest concern is sloppy research practice, such as running several experiments and only reporting the ones that work.

It's true that fraud is not the major issue, as I and many others have said. But bad practice, such as p-value fishing, is in no way "sloppy" as Shanks says. Running multiple experiments to get a positive results is a sensible and effective strategy for getting positive results; that's why so many people do it. And so long as scientists are required to get such findings to get publications and grants, it will continue.

Behavior is the product of rewards and punishments, as a great psychologist said. We need to change the reinforcement schedule, not berate the rats for pressing the lever.

Earlier, Shanks writes that evidence of unconscious influences on human behaviour - a popular topic in Stapel's work and in social psychology generally -

is easily obtained because it usually rests on null results, namely finding that people's reports about (and hence awareness of) the causes of their behaviour fail to acknowledge the relevant cues. Null results are easily obtained if one's methods are poor.

Thus journals have in recent years published extraordinary reports of unconscious social influences on behaviour, including claims that people are more likely to take a cleansing wipe at the end of an experiment in which they are induced to recall an immoral act [etc]...

...failures to replicate the effects described above have been reported, though often papers reporting such failures are rejected out of hand by the journals that published the initial studies. I await with interest the outcome of efforts to replicate the recent claim that touching a teddy bear makes lonely people more sociable.

Here Shanks first says that null results can easily result from poorly-conducted experiments, and then criticizes journals for not publishing null results that represent failures to replicate prior claims! But null replications are very often rejected because a reviewer says, like Shanks, "This replication was just poorly-conducted, it doesn't count." Shanks (unconsciously no doubt) replicates the problem in his article.

So what to do? Again, it's a systemic problem. So long as we have peer-reviewed scientific journals, and the peer-review takes place after the data are collected, it will be open to reviewers to spike results they don't like - generally although not always null ones. If reviewers had to judge the quality of a study before they knew what it was going to find, as I've suggested, this problem would be solved.

Other people have great ideas for fixing science of their own. The problem is structural, not a failing on the part of individual scientists, and not limited to social psychology.

My Breakfast With "Scientism"

One morning, I awoke convinced that science was the only source of knowledge. I had developed a case of spontaneous scientism.

The first challenge I faced was deciding what to eat for breakfast. Muesli, or cornflakes? Which would be the more scientific choice? I decided to go on the internet to look up the nutritional value of the different cereals, to see which one would be healthiest.

My computer was off. So first I'd need to turn it on - but how? From past experience, I suspected that pressing the big green power button on the front would do it - but then I remembered, that's merely anecdotal evidence. I needed scientific proof.

So I made a mental note to run a double-blind, randomized controlled trial of "turning my computer on" tomorrow.

Lacking nutritional data, I decided to pick a cereal by taste. I like muesli more than cornflakes. At last, a choice! Muesli it is, I thought - until I realized that I didn't actually know which one I preferred more. I had a gut feeling I liked muesli, but that's not science. What if, in fact, I hated muesli? Science couldn't tell me, at least not yet.

Another mental note: conduct cereal taste preference study, day after tomorrow. No breakfast for me, today.

By now, I was hungry, confused and annoyed. "This is getting ridiculous!", I tried to exclaim to no-one in particular - but then I realized - I could not even speak because I knew next to nothing scientific about the English language.

Sure, I had vague intuitions about how to put words together to express meaning, but that's just unscientific hearsay that I'd picked up as a child (no better than a religion, really!) In order to communicate, I'd need to study some proper science about semantics and grammar... but, oh no, how could I even read that literature?

Faced with the impossibility of doing anything whatsoever purely guided by science, I decided to go back to bed... yet with no scientific basis for controlling my own muscles, I collapsed where I stood, bashing my head on the breakfast table as I fell. 

Luckily, the bump on the noggin cured me of my strange obsession, and I lived to tell the tale.

---

Many people will tell you that "scientism", the belief that science is the only way to know anything, is a serious problem, a misunderstanding that threatens all kinds of nasty consequences.

It's not, because it doesn't exist - no-one believes that. If they did, they would end up like the unfortunate narrator in my story.

Everyday, we make use of many sources of information, from personal experience and learning to simply looking at things, whether they be right in front of your eyes or on TV. This is knowledge, and no-one thinks that we ought to replace it with "science", if that were even possible.

"Scientism" is a fundamentally unhelpful concept. Scientists are often wrong, and sometimes they're wrong about things that other non-scientists are right about. But each such case is different and must be judged on its own merits.

The Small World of Words

I've been asked to encourage people to take part in an online psychology study called The Small World of Words

I get a lot of this, and I usually don't respond to such requests, but this one looks pretty interesting.

The project aims to collect the world's biggest word association database. You see a series of words and you just have to type in the first three words that pop into your head.

Here's some more about it:

On average, an adult knows about 40,000 words. Researchers in psychology and linguistics are interested in how these words are represented mentally. In this large-scale study we aim to build a network that captures this knowledge by playing the game of word associations. You can help us with this project by participating in this short and fun study.

The study consists of giving the first three words that come to mind for a list of 14 items.
All ages and nationalities are welcome, but please note that we do require all participants to be fluent English speakers.

It's the sheer scale of this that makes it cool. They've got some 60,000 participants, and over a million associations already, but they're aiming for almost 300,000 people - which would make it not just the biggest word association study ever, but the biggest psychology study of all time, as far as I know.

It only takes about 2 minutes to complete and it's actually quite revealing. Out of 14 words I managed to associate a full 3 of them with 'pain', which disturbed me somewhat.

So take a look and spread the word (associations)...

Am I Attacking Neuroscience?

A New York Times article just out says:

Neuroscience: Under Attack

Under attack by who?

Er... me. And the rest of the usual suspects:

A gaggle of energetic and amusing, mostly anonymous, neuroscience bloggers - including Neurocritic, Neuroskeptic, Neurobonkers and Mind Hacks - now regularly point out the lapses and folly contained in mainstream neuroscientific discourse. 

I had promised not to do any more self-referential posts, but this one wasn't my fault. Just when I thought I was out, they pull me back in.

Anyway, I'm pretty happy with how Neuroskeptic's presented in the article, but not entirely.

The headline is sensationalist - I don't see myself as attacking neuroscience and I don't think any of the others do either. We are trying to defend neuroscience against errors and misrepresentations. My ideal is The Sceptical Chymist, where skepticism helped, rather than undermined, chemistry.

But the job of a headline is to be sensationalist so that's OK. Most of the piece is very good. I'm all on board with this:

Meet the "neuro doubters". The neuro doubter may like neuroscience but does not like what he or she considers its bastardization by glib, sometimes ill-informed, popularizers.

Yet I can't quite go along with this:

A number of the neuro doubters are also humanities scholars who question the way that neuroscience has seeped into their disciplines, creating phenomena like neuro law, which, in part, uses the evidence of damaged brains as the basis for legal defense of people accused of heinous crimes, or neuroaesthetics, a trendy blend of art history and neuroscience.

Admittedly this wasn't directly aimed at me because I'm not a humanities scholar, but I believe that neuroaesthetics and neurolaw are absolutely valid - in theory.

I'm not defending any particular manifestation of those, and I've criticized quite a few. But in the abstract, I see nothing wrong with neuroscience helping to explain those things. It will be difficult in practice, but it's fine to try.

Brain Mnemonics

Learning your way around the brain is pretty difficult; I spent a long time trying to learn the basics of neuroanatomy... and I bet if you did a pop quiz of neuroscientists the results would be rather embarrassing.

Over time, I've come up with mnemonics to help me remember the trickiest points. Here's some of my favourites.

Gyri and Sulci - which is the ridge, and which the trough? A sulcus sounds like sulk which is what you do when you're feeling low - sulci are the troughs.

Dorsal and ventral - which is lower? Ventral, because the V in ventral is an arrow pointing down.

Axial, Coronal and Sagittal sections - Sagittal is the only kind of cut that could cause Split-brain Syndrome by cutting the brain into two Symmetrical halves.

Axial is the section you'd get if you drew a line across your the face from left to right (and then extend that back), or in other words around your head.

If you have any other good ones, please share them in the comments.

Are Gay Men Happier?

A neat little study from UCLA psychologists Francisco J. Sánchez and colleagues examines the mental health of homosexual men using a unique identical twin design.

The paper kicks off with a remarkably lucid introduction:

Men would rather drive around lost than stop and ask for directions. Although this is a gross stereotype, the notion that men should be self-sufficient and able to solve their own problems is a dominant ideal within traditional views of masculinity... men who rigidly adhere to such ideals may harm their own health if they avoid seeking help when they need it.

In general it reads more like a blog post than an academic paper, which is great. If all papers (and especially social science ones) were written this way, I would be a much happier man. Speaking of happy men...

The authors' basic idea is that many men, wishing to appear 'manly', don't talk about or get help for their problems, especially psychological issues: boys don't cry, and men certainly don't. However, the authors argue that gay men, generally less encumbered by traditional masculinity, may be an exception to this rule.

So they took 38 pairs of male 'identical' twins, who grew up together, but who weren't quite identical: one of each was gay, and one straight. By controlling for most genetic and environmental factors, these twins provide a kind of natural experimental test of the effects of homosexuality per se. Not a perfect one, but about as good as we're going to get.

In accordance with the authors' predictions, gay men were indeed more open to seeking psychological help.

But unexpectedly, they were actually less likely to report experiencing psychological distress (on this scale). That's surprising, given several previous reports of higher rates of mental illness in homosexuals, which has been dubbed 'velvet rage'.

Sánchez et al's data suggest that gay men may be, er, more gay (...the other kind), and that their increased rates of diagnosed mental illness are a product of their greater willingness to seek help: maybe straight men are just in denial.

But there's a lot of caveats here. It's a small study, based purely on self report measures, and the gay twins were compared to their own straight twins, but those twins are quite possibly not typical of straight men in general. It might also be that having a straight twin makes life easier for gay men. Still, it's an interesting set of data.

Sánchez FJ, Bocklandt S, and Vilain E (2012). The Relationship Between Help-Seeking Attitudes and Masculine Norms Among Monozygotic Male Twins Discordant for Sexual Orientation. Health Psychology PMID: 23025300

The Two Problems With Science

There's lots of concern at the moment over mistakes, misconduct and misbehaviour in science.

This concern is a good thing. There are serious, systemic problems with modern science as I and many others have long argued.

However, I worry that much of the recent discussion has failed to distinguish between two fundamentally distinct problems. On the one hand, we have outright fraud - i.e. making up data, or otherwise lying, breaking the basic rules of science.

On the other hand we have questionable practices such as: publication bias, p-value fishing, the File Drawer, sample size peeking, post-hoc storytelling, and all of the other dark arts that can lead to false positive science. These are permissible, even encouraged, by the current rules of doing and publishing science.

These two problems are similar in some ways - they're both "bad science", they both lead to failures to replicate, etc. - but in underlying essence they're very different, so much so that I'm not sure they can be usefully discussed in the same breath.

Fraud and questionable practices are different in terms of their harms. Fraud is a more serious act and it causes local harm, introducing major errors into the record. But in terms of its overall effects, I believe questionable practices are worse, as they systematically distort science: ensuring that, in some cases, it is difficult to publish anything but errors.

Fraud and questionable practices call for different solutions. Broadly speaking, fraudsters break the rules, so to stop them we need to enforce those rules, via deterrence, detection, and punishment - like with any criminal act. With questionable practices, it's the opposite: here the problem is the rules (or the lack of them), and the solution is to reform the system.

It's been suggested that fraud and questionable practices share a common cause in the "pressure to publish", the "publishing environment", the "culture" of modern science etc. But while this is a good explanation for questionable practices, I don't think this can explain fraud, any more than, say, the desire for money can explain theft.

Yes, thieves desire money, and yes they steal in order to get money, but everyone else wants money as well, yet most of us don't steal, so that's not an explanation. Frauds fake data to produce publications. But all scientists are under pressure to produce good publications and they always have been - which is why fraud is not new - what's changed recently is the criteria for a 'good' publication.

Now in retrospect, I blurred these distinctions somewhat with my own 9 Circles of Scientific Hell, in which I placed 6 questionable practices and 2 forms of misconduct on the same scale of "sinfulness". In fact there are two distinct hierarchies. In my defence though, that was a cartoon.

I think finance offers a great analogy here.

In finance, you have some people who break the rules. Bernie Madoff is the current poster boy for this. Such people harm others by outright criminal acts. But then we have the people who play by the rules, and still cause harm. The global financial crisis was in essence caused by all of the major American banks going all-in on a bet, and losing. Yet no-one broke the rules: the regulations allowed banks to gamble. The problem was not rule-breaking, but the rules (or lack thereof).

Here's the curious thing: the financial crisis did more harm than Madoff's scam, even though what Madoff did - theft by fraud - was more immoral than what the bankers did - gambling unwisely.

That's confusing to our ethical sense and our emotions (who should we feel more angry at? Who's 'worse'?) but it's really no surprise: precisely because what the banks did was above board, everyone did it so the damage was huge. If it had been illegal for banks to gamble all their money at once, individual banks might still have broken that rule, locally, but it's unlikely that the system would have been threatened.

Maybe you can see where I'm going with this: everyone following bad rules is often worse than individuals breaking good rules.

Science has its share of fraud. Hauser, Smeesters, Fujii - they broke good rules against such deceit. They are the Bernie Madoffs of science. But then there's 'questionable practices' like publication bias, p-value fishing, the File Drawer, and all the rest, which are allowed, but which are universally acknowledged to be bad for science. Scientists using these dark arts (and I don't know any who never do) may be the Lehman Brothers of science.

Science: Growing Too Fast?

There's a widespread perception among scientists that we're living in an era of relentless growth in terms of the number of scientific papers being published.

Many say that quantity has increased at the expense of quality: people are publishing "any old rubbish" or splitting their work into as many papers as possible, driven by the publish-or-perish culture of modern academia.

But is this true? To try and find out, I looked at the number of papers published each year, in English, on PubMed, for the past 30 years.

Here's the data: it shows an increase in the number of papers coming out each year, except for a small negative blip around the year 1997:

Now, when I first eyeballed this curve, I got the impression that growth has accelerated recently, consistent with the "recent pressure to publish" idea.

But here's the same data with each year's publications expressed as a ratio to the previous year's:

This reveals that the relative annual growth in the number of papers published has actually been pretty constant over the past 30 years. It's generally been around 4% (ratio of 1.04), and almost always within the range 2% to 6%. In other words, every year, scientists publish the same number of papers they did last year, plus about 4%.

The past few years have not seen especially strong growth, relatively speaking. At most we can say that year-on-year growth has been at the upper end of the historical range, 4 to 6%, but that's no faster than in the 1980s.

Still, this is a lot of growth. Assuming that it stays at 5% year on year, we'd expect a million new papers published in 2016, and two million in 2030.

But is that really feasible? Is there any good reason that science should grow exponentially in this way? Can that continue, or will we reach "peak science" or at least a plateau?

The Rise of Science Spam

I'm not sure if it's just me, but in the past few months I've been getting an inordinate amount of scientific spam.

This is at my real-world email address, under the name I publish my papers. I can only think that some nefarious hucksters are trawling scientific journals and harvesting contact details from the author lists. Either that, or a legitimate organization I've signed up to has shared their mailing list; but a robot seems more likely.

Whatever's going on, it's getting worse, at least for me. A few months ago, I got maybe one piece of sci-spam a week, which was tolerable. Now it's up to half a dozen or more per week and getting ridiculous. Here's what I've got just in the past 10 days.

Lab Products

These I can kind of see the point of: if you have a product to sell, you want to advertise it to people who might want to buy it. The problem is that as someone who scans brains, and last touched a pipette about 8 years ago, I really don't want to buy:

Two readouts for the price of one - Calcium and Arrestin. Select Any Gq-Coupled Calcium Cell Line and Save! Special Fall Savings!

Nor am I interested in:

Dear Colleague, Through your publications, we have noted that our Catalase antibody (GTX110704) may be useful to your current work. Catalase is a peroxisomal enzyme that converts hydrogen peroxide to oxygen and water, making it a key feature of the cell’s defense against oxidative stress...

To be honest, if you need to be told what catalase is, you're unlikely to want to buy a product that will let you measure it... but that's just my opinion, and I'm not a spammer.

Conferences

This is where it gets weird. Spam inviting people to conferences, doesn't make sense, because by definition, if a conference is resorting to spam to get attendees, it's not very good. Respected conferences are often oversubscribed; the whole point about a conference is that people want to be there, because other interesting people are expected to be there. Reputation is everything.

 Here's a few I've been offered in the past two weeks:

31 October 2012 is the deadline to save on registration for APAL 2012. Covering current and specialty aspects of mental health, this is one gathering that you cannot afford to miss! ... Please note that payment can be made by credit card.

and...

1st International Conference on Cultural Psychiatry in Mediterranean Countries, Tel Aviv, Israel | 5-7 November, 2012 The countdown has begun: Less than 2 months to WPA-TPS in Tel Aviv.

and...

Dear colleague, I welcome you to our first international conference on the topic of Integrated psychiatry and clinical psychology as our valued guest speaker. Because of your publication profile, we invite you to present ideas related to your works on related to the topic of the conference. Psychiatry and clinical psychology branches have evolved rapidly in past couple of decades. So much so that several sub-branches have emerged as parts of these sciences. The theme of present conference is integrated psychiatry and clinical psychology...

Journals

If you are resorting to spam to get to people to write for your journal, I don't ever want to read it and will never cite anything published in it.

Even if you were only angling for readers, I'd be suspicious of your integrity, but to spam for people to submit to you is absurd. Even mediocre journals nowadays get far more submissions than they can ever print. So if your journal isn't even mediocre enough to attract people then you have a real problem.

Journal of Anesthesiology and Clinical Science (ISSN 2049-9752)

Journal of Anesthesialogy [sic] & Clinical Science is an Open Access and peer reviewed Journal which aims to publish top quality papers on administration of anaesthesia during surgeries and pain management etc. The Journal has well [sic] established Editorial Board and follows rigorous peer review for all the [sic] manuscript's [sic]. visit [sic] the Journal to find latest [sic] articles published... We invite you to submit your research work/paper for the Anesthesialogy and Clinical Science Journal...

Oh, and you also have a problem if you can't spell your own journal title nor write coherent English despite claiming to only accept high quality scientific papers in that language.

Has anyone else noticed a surge of this kind of thing recently?

Publication Bias in Animal Research

Publication bias has historically been thought of mostly in the context of clinical trials. But I have been banging on for the past 4 years about how it's a problem for more 'basic' science as well.

I'm not alone in my concerns as an interesting new paper reveals: Publication Bias in Laboratory Animal Research. The authors surveyed the approximately 3,000 Dutch scientists involved in research on laboratory animals. The response rate was about 20%.

When asked how much animal research ends up being published, university researchers estimated about half, but industrial scientists put it at only about 10% - which, if true, suggests that publication bias in Pharma animal work is extremely serious.

In terms of solutions, the survey considered two ideas which Neuroskeptic readers will be familiar with - public pre-registration of studies:

Mandatory anonymous publication of research protocols of all ethics-approved animal research experiments in a publicly available database

and also open access to all data:

Mandatory anonymous publication of a brief structured form in a publicly available database, that gave main results or explained why an experiment could not be completed

On average the surveyed researchers felt that these measures would aid scientific progress; improve the validity of the literature; and prevent wasteful duplication of effort - but they also worried that it would increase bureaucracy.

Now, bureaucracy is second only to bias on my list of Things I Hate About Science, so I share their concern - but I really think registration wouldn't have to involve any extra paperwork. In many cases, it could be implemented simply by making existing data public.

For instance, grant applications, and requests for ethical approval, already contain detailed a priori protocols in most cases. They could so easily be published (perhaps with certain details removed for confidentiality reasons) and turned into a powerful weapon against publication bias.

Having said that though - it easily could end up being needlessly complicated and obstructive, as so much of the scientific process unfortunately is today. It will all depend on how it's implemented.

This is why I think it's so important that, as scientists, we reform science ourselves, and get it right, rather than leaving it to the bureaucrats, who won't.

Ter Riet G, Korevaar DA, Leenaars M, Sterk PJ, Van Noorden CJ, Bouter LM, Lutter R, Elferink RP, and Hooft L (2012). Publication bias in laboratory animal research: a survey on magnitude, drivers, consequences and potential solutions. PloS one, 7 (9) PMID: 22957028

What Is Science?

The other day I was in a discussion about what "science" is. I've written about this before but this debate got me thinking about it again and I thought I'd set out what I think in more detail.

I wasn't sure how best to structure this so I'm going to start with my main claim, followed by a Q and A bit. The Q's are not intended to be straw-men or caricatures, they're questions I've asked myself in the course of thinking about this.

My Claim: "Science" is just the process of looking at the world and thinking about the evidence in an effort to understand it. It's not a special form of knowledge, scientists don't use a special 'scientific method' - scientists just look and think about things. They may use special equipment and techniques, but in essence it's no different to what we all do every day. As such it makes no sense to talk about the 'limits of science' or 'what science can't tell us', unless by that we mean the limits of human knowledge itself, because science just is knowledge.

Q: "But if science were just observation, everyone would be a scientist and it becomes meaningless." - No: for the same reason that not everyone is a poet, even though anyone can write a poem.

Science is observation informed by previous scientific findings i.e. it is expert observation. Anyone can, say, look at the stars but this doesn't make them an astronomer. Astronomy is in essence just looking at the stars and thinking about them - in a broad sense - but to contribute to astronomy, you first need to know the relevant background, which few except astronomers do.

Likewise, anyone can write a poem, but few of us can make a living out of it.

Q: "OK, but still, if science is just observation, then all forms of knowledge are science." - This is a tricky one, but the answer is crucial to understanding my point.

A few hundred years ago, the word "science" in English did indeed just mean "knowledge". However, more recently, it has come to mean a particular subset of knowledge: roughly, it today includes physics + chemistry + biology. Maybe some others.

We lump these three (or more) things together and call the lump "science". But this lump is more or less arbitrary. Physics + chemistry + biology don't share a special essence, which sets them apart from other kinds of knowledge (other 'sciences' in the older sense.)

So I'm not saying that all knowledge is "science" in the modern sense. The modern word "science" only includes a limited portion of knowledge. But I am saying that the rest of knowledge is essentially no different from science, because "science" is just an arbitrary subset of knowledge.

Here's a picture of what I mean:

Or here's an analogy. The color spectrum has infinite different shades. We conventionally divide it up into "red", "orange", "yellow" etc. and that's fine for most purposes. But there is no essential difference between "orange" and "yellow" and no clear dividing line: they are just collections of shades.

Science is a colour of knowledge. It's not a true kind.

Q: "But if the difference between science and knowledge is arbitrary, are you saying that The Scientific Method is the only way to knowledge?" Not at all. I don't think 'The Scientific Method' exists.

This follows from the fact that "science" is an arbitrary lump. Scientists are a diverse bunch and they use many different methods. Theoretical physicists, for example, use methods which are very close to those of mathematicians - who are not 'scientists' by most definitions. Zoologists use others, and you can get by in (most) of zoology without knowing any math at all. And so on.

In fact, there's almost as much difference between branches of "the same science" as there is between sciences. Just like "science", "biology" is a lump of diverse things, although not quite as arbitrary a lump.

Outside science, people use all kinds of methods as well. Historians have their set of methods, economists have others, all tailored to the particular demands of the case. That's exactly how it should be - all knowledge comes from observation and, to observe different things, you need different methods.

There are many different kinds of facts, but a fact is a fact, whether it's a scientific fact, a historical fact, or just an everyday fact. "Shakespear wrote Hamlet" is just as true as "The earth orbits the sun" is just as true as "It's raining" (if it is, in fact, raining.) The facts of history are just as true as the facts of biology.

But there is a grain of truth in the question here: I am saying that observation is the only way to knowledge.

Q: "That's very simplistic. Are you saying that we can only know what we can see and measure?" - No because I'm using "observe" in the broadest sense here, to include things like noticing, sensing, feeling, seeing, hearing, being told that, reading about...

You observe that it's raining: maybe you look out the window, maybe you hear the rain on the roof, maybe someone comes in from outside dripping wet. You observe that you feel hungry. You observe that Obama is president (even if you've never actually seen him) by watching the news. Etc.

"Observe" is in many cases an awkward word and I'm not saying we should use it in all those cases. My point is that these are all ways of finding stuff out about the world. In that broad sense, I think all knowledge comes from observation - although maybe indirectly, through thinking about observations; thoughtless observation is almost as unhelpful as purely abstract speculation.

Of course, it's easier said than done: very often it's not clear what we've observed (is it real or some kind of trick, mistake, illusion?), what it means, whether it matters, or even what the question is. This is why there's always room for debate, controversy and doubt, at least at first, before all the observations have been made.

Q: "But what you're saying is merely trite and obvious: 'science is based on observation' 'there is no one single scientific method', well, duh!" - From a certain perspective, they are obvious, but if you think so, then you ought to agree with everything else I've written here. They're all connected, you can't take some and leave the rest.

Replication Alone Is Not Enough

Psychology has lately been hit by high-profile fraud scandals, and broader concerns over questionable research practices. Now the Society for Personality and Social Psychology (SPSP) has released a statement on "Responsible Conduct", and a task force has produced a report.

This is a start, and the SPSP is to be commended for facing up these problems (which affect many other fields) relatively early. However, neither of their documents contains much meat in my view.

Point One on the task force report is that "Replication is the key to building our science" and they suggest a "web site for depositing replications and failures to replicate" - but don't mention that various enterprising researchers have already made one. Nor do they tip their hats to the Open Science Initiative addressing just this issue. This makes me worried that they're planning to reinvent the wheel.

More fundamentally I disagree that replication is key to psychology or any field. Our goal should be replicability. Failure to replicate findings is a symptom of problems with those original findings, rather than being a problem in and of itself. Good results replicate; we want better results to be published.

In other words, we should strike at the root cause of invalid research, namely, the perverse incentives towards publishing as many eye-catching positive results with p values below 0.05 as possible by any means necessary. P-value fishing, selective reporting, post-hoc "prior hypotheses" and other questionable practices are a large part of what make unreplicable results.

We should encourage replication, but it's no panacea.

An overemphasis on replication, without addressing the incentives, could actually harm science. It could lead to scientists spending all their time worrying about the political drama of who's replicating who and why, and which questionable practices they can use to replicate their friends' data - rather than actually doing science.

This is why we shouldn't be satisfied with any reform effort that puts replication before replicability. If you can fudge a result, you can fudge the data a replication. How to fight questionable practices is another question but I've proposed reforms that I think would work, namely pre-registration of hypotheses, methods, and statistical analyses. Others have their own ideas.

A lesson from clinical medicine here. Clinical trials of new drugs adopted pre-registration, but only after they tried replication and it didn't work. Pharmaceutical regulators have long required multiple demonstrations of drug efficacy. One trial was not enough. Sounds good - but the problem was that drug companies just did lots of trials and analyses, picked the positive ones, and used them.

So in summary: replication is important, and we don't do enough of it, but replication alone is not enough to fix psychology.

Social Science and Language, Again

On Sunday I asked, Why Don't Social Scientists Want To Be Read? I accused much of social science of using unnecessarily complex jargon.

This post prompted many excellent comments - including responses on other blogs e.g. Andy Balmer and Graham Davey.

The most common argument against my post was, in essence: Every science has a specialized, technical vocabulary. You wouldn't criticize a neuroscience abstract for being inaccessible to a layperson, so it's unfair to expect that from sociology.

This is a good and convincing point. Yet I think that, on closer inspection, it relies on some rather major assumptions.

The natural sciences do have a 'specialized' vocabulary, but only because they deal with things that are of special interest. What is 'special' or 'technical' about the word forebrain (to borrow an example from Andy Balmer) is merely that only neuroscientists are interested in the object, forebrains. It's not part of the everyday English language, because it's not part of everyday life.

There's nothing inherently 'academic' about forebrain, in other words. Plenty of similar terms like 'forearm' and 'foreskin' are part of everyday English, not because they're somehow less precise or less formal, but just because they crop up more often.

Everyday English is inadequate for natural science because scientists study things outside everyday experience. But the major object of the social sciences is everyday human life. Social scientists are interested in things that everyone is interested in - why people think, feel and behave the way they do.

So if the social sciences have need of a technical vocabulary, in the same way as the natural sciences, this would imply is that our everyday language is fundamentally inadequate to understanding the everyday world - in other words, that it is just inadequate, period.

Everyday English has developed to allow people to talk to one another. And the main thing people talk about, is one another, i.e. about society. Like any other branch of science, the social sciences need a rich vocabulary to describe all of the things they study: but don't they already have one - English?

Maybe not. It may be that ordinary English can't express the truth about society. But if you take that seriously, that's a pretty radical claim, akin to saying that the great majority of people are in the dark about how the world works. It's much more radical than saying that chemistry or neuroscience needs special words.

To be clear, I'm not making the populist argument that "Social science is all rubbish - the average man in the street knows better than these eggheads!" The average person is wrong about all kinds of important things, but I suspect that they're sometimes in the right ballpark, as it were, and that their errors are not a matter of lacking the proper words.

Why Don't Social Scientists Want To Be Read?

Here's the abstract of a paper just out called In pursuit of leanness: The management of appearance, affect and masculinities within a men's weight loss forum.

In a somatic society which promotes visible, idealized forms of embodiment, men are increasingly being interpellated [sic] as image-conscious body-subjects. Some research suggests that men negotiate appearance issues in complex and varied ways, partly because image concerns are conventionally feminized. However, little research has considered how overweight men construct body projects in the context of weight loss, or how men talk to each other about weight management efforts. Since sources of information and support for overweight men are now provided online, including dedicated weight loss discussion forums, our analysis focuses on one such forum, linked to a popular male-targeted magazine. We conducted a thematic analysis of selected extracts from seven threads on the forum. Our analysis suggests a widespread focus on appearance, as well as the use of emotion categories when describing difficult bodily experiences. Invariably, however, such talk was carefully constructed and constrained by hegemonic masculinities founded on discipline, work-orientation, pragmatism and self-reliance. The findings are discussed in relation to magazine masculinities and aesthetics, as well as literature on male embodiment.

Phew. Now I think it's fair to say that this is a typical example of what might be called the "social sciences style" of writing. That's why I've chosen to blog about it; nothing I'm going to say is a criticism of this paper as such, but rather of the whole genre.

Why do social scientists write like this?

This paper is about a really interesting topic - the mixed messages men get about what it means to be "a man" or "manly" in today's society. Very topical, not at all 'niche', and important in lots of ways. So why is it written in a way which makes it impenetrable to all except specialists?

I don't think it has to be that way. I've rewritten this abstract, and I've tried to say the same thing without the jargon:

Modern men face a dilemma: society tells them that they ought to have an attractive body, but they are also warned that being concerned about beauty and body image is a feminine trait. However, little research has considered how overweight men think and talk about weight loss. Online weight loss forums offer a window onto such issues, so we analyzed seven threads from one such site, linked to a popular men's magazine. We found that while men took part in (often emotional) discussions of their own appearances and bodies, they always framed such talk strictly within conventionally "masculine" terms such as discipline, work-orientation, pragmatism and self-reliance. We discuss this, in the context of men's magazines treatment of masculinity and male beauty, and relate this to previous work.

Whether I've succeeded, I'll leave others to judge, but I think I have and there's no trick to it - I just read the original, tried to understand it, and wrote down my thoughts.

I'm not saying that the original abstract was "badly written". I suspect it was quite expertly written but that the purpose of writing it was less to communicate ideas clearly, than to satisfy some set of criteria of what 'serious social science' should be like.

If I'm right - isn't that a shame? The ideas here deserve a wide audience, so why aren't they aimed at one?

Bennett E, and Gough B (2012). In pursuit of leanness : The management of appearance, affect and masculinities within a men's weight loss forum. Health (London, England : 1997) PMID: 22815334

Catching Fraud: Simonsohn Says

Everyone's been talking about psychologist Uri Simonsohn and his role in the downfall of two scientific fraudsters.

When the story first broke, the methods Simonsohn used that allowed him to spot the dodgy data were mysterious - which only added to the buzz. The paper revealing the approach is now up online and it's a must-read. It's not often a statistics paper offers the train-wrecky schadenfreude of watching two fraudsters' careers come to a well-deserved end.

What's rather disturbing about the article, however, is that it doesn't really contain much that's new, in principle. Simonsohn used statistics to spot data in published papers that was, in effect, 'too good to be true'. He then followed up seemingly dodgy cases with some more stats, using simulations of what real data ought to look like, to verify that it was in fact made up. A simple idea in retrospect but one that's never been tried before. I don't think there's a single "Simonsohn method", rather, the paper uses multiple techniques, each one tailored to the particular data in question.

But it shouldn't have come to this. Someone else ought to have spotted that the data looked dodgy.

Take this table from one of Simonsohn's conquests, a soon-to-be-retracted paper by Lawrence J Sanna et al:

We now know that the data from Studies 2,3 and 4 were all made up. Each study compared 3 conditions, and what makes these data dodgy is that the standard deviations of the 3 sets of results for each study were almost identical. The chances of that happening are very low and it suggests that someone has (clumsily) made the data up.

I'm going to say that these data are obviously suspicious, at least to anyone who has worked with real data. Maybe you'll say that hindsight is 20/20, but Simonsohn didn't need hindsight and the stats he used were nothing remarkable. I'm not saying that to belittle his achievements, he deserves plenty of credit. But other people deserve blame.

Namely, whoever peer reviewed this paper should have spotted that these data looked unusual - and they should not have needed any special statistical tools to do so.

Simonsohn calls for journals to require that the raw data be made available for all published work, on the grounds that. That's a great idea - and not just because it would help catch bad science: it would facilitate proper research and teaching no end. But Simonsohn didn't need the raw data to detect these cases of fraud - he only checked the raw results to confirm the suspicions based on the published data.

Checking that the data are valid is the job of peer reviewers, and they dropped the ball. Instead Simonsohn had to conduct his own private crusade against fraud... a bit like Batman. Batman is awesome, but the point about Batman is that he's only needed because the police can't or won't cope on their own. He's not a superhero, he's just a guy with the will.

Peer reviewers are the police of science, but all too often, they're asleep on the job. Not just in psychology. Retraction Watch provides plenty of examples of published results in biology that were faked, often in comically crude fashion, and should have been obvious to anyone paying attention.

Peer reviewers are usually anonymous. I wonder if a policy of retrospectively naming and shaming the reviewers when a paper turns out to have been fraudulent, might help motivate them...?

B. F. Skinner vs. the Rorschach Test

What happened when the world's most no-nonsense psychologist took a Rorschach test?

A fun little paper reports on B. F. Skinner's Rorschach results. He agreed to be tested as part of a 1953 project psychoanalysing various eminent scientists. The scientists were anonymous at the time but now Norwegians Cato Grønnerød et al have dug them out of the archives (Skinner has been dead since 1990).

Skinner was the world's leading exponent of behaviourism, a school of thought that held roughly that it's impossible to know anything about "inner" mental states or thoughts, and that they might not even exist, so all we could do was look at and try to predict behaviour (edit: see comments for clarification).

It was never an especially convincing idea to be honest and behaviourism is now pretty much dead although many of the techniques pioneered by Skinner live on in the form of tests on lab animals to determine the addictiveness of drugs and so forth.

But in the mid-20th century it was very popular and Skinner was a well-known figure, the Jonah Lehrer of his day in many ways although rather more controversial.

Anyway. Grønnerød et al report that when Skinner was asked to describe those famous inkblots -

The most evident feature of the protocol is the huge number of responses, showing a highly productive and creative person. But complexity is sacrificed for quantity... No perceptual distortions are evident, and reality testing and ability to function neutrally are in place. We found no signs of cognitive distortions, although some responses have an idiosyncratic twist... He might be an assertive person with a tendency to view relations as generally competitive and an area for the expression of his own needs, rather than an area of mutual support and belonging.

Although he shows an interest in others, the balance between real and whole humans and other human representation suggests that perception of self and others is based more on fantasies and wishes than on real-life perceptions... “Necrotic looking,” “wounded animal,” and “sheep pushing the two wolves away” might reflect projected aggression. These processes point to more primitive defense mechanisms...

Which is exactly the kind of speculation that Skinner spent his career trying to put a stop to. Still, it's an interesting paper, although I think it tells you more about the Rorschach than about Skinner.

Grønnerød C, Overskeid G, and Hartmann E (2012). Under Skinner's Skin: Gauging a Behaviorist From His Rorschach Protocol. Journal of personality assessment PMID: 22731841