Drunk Rats Could Overturn Neurological Orthodoxy

A form of brain abnormality long regarded as permanent is, in fact, sometimes reversible, according to an unassuming little paper with big implications.

Here's the key data: some rats were given a lot of alcohol for four days (the "binge"), and then allowed to sober up for a week. Before, during and after their rodent Spring Break, they had brain scans. And these revealed something remarkable - the size of the rats' lateral ventricles increased during the binge, but later returned to normal.

Control rats, given lots of sugar instead of alcohol, did not show these changes.

This is really pretty surprising. The ventricles are simply fluid-filled holes in the brain. Increased ventricular size is generally regarded as a sign that the brain is shrinking - less brain, bigger holes - and if the brain is shrinking that must be because cells are dying or at least getting smaller. So bigger ventricles is bad.

Or so we thought... but this study shows that it might not always be true: alcohol reversibly increases ventricular volume over a timescale of days. It does so, the authors say, essentially by drying brain tissue out; like most things, if you dry the brain out, it gets smaller (and the ventricles get bigger) but when the water comes back to the tissues, it expands again.

As you can see here in Figure 2...

Maybe. I admit that just eyeballing this, it looks more like the ventricles are getting brighter, rather than bigger, but I'm not familiar with the details of water scanning. Maybe some readers will know more about it.

If it's true, this is big - maybe it's not just high doses of alcohol that does this. Maybe other drugs or factors can shrink or expand, the ventricles, or even other areas, purely by acting on tissue water regulation, rather than by anything more 'interesting'.

Take the various claims that some psychiatric drugs boost brain volume while others decrease it, just for starters...could they be headed for a watery grave?

Of course, this is in mice - and it might not translate to humans... we need to find out, and I for one am keen to apply for a grant. Here's my draft:

Participants: 8 healthy-livered neuroscientists.
Materials: 1 MRI scanner, 1 crate Jack Daniels.
Methods: Subjects will confer to pick a Designated Operator, who will remain sober. If no volunteers for this role are forthcoming, selection will be randomized by Bottle Spinning. All other participants will consume Jack Daniels ad libitum, and take turns being scanned. Once all Jack Daniels is depleted, participants will continue to be scanned until fully sobered up (defined as when they can successfully spell "amygdalohippocampal").
Instructions to Participants: i) what happens in the magnet, stays in the magnet. ii) If you 'dirty' the scanner, you clean it up. iii) Bottle caps are not MRI safe!

Er... seriously though, someone should check.

Zahr NM, Mayer D, Rohlfing T, Orduna J, Luong R, Sullivan EV, and Pfefferbaum A (2013). A mechanism of rapidly reversible cerebral ventricular enlargement independent of tissue atrophy. Neuropsychopharmacology  PMID: 23306181

Smart People Say They're Less Depressed

The questionable validity of self-report measures in psychiatry has been the topic of a few recent  posts here at Neuroskeptic.

Now an interesting new study looks at the question in issue from a new angle, asking: what kind of people report feeling more or less depressed? Korean researchers Kim and colleagues found that intelligence and personality variables were both linked to the tendency to self-rate depression more severely.

The study involved 100 patients who'd previously suffered from an episode of depression or mania and who, according to their psychiatrist, had now recovered and were back to normal. Kim et al looked to see what the patient thought about their mood, by getting them to complete the Beck Depression Inventory (BDI) self-report questionnaire.

This was compared to the clinican-administered HAMD scale (another Neuroskeptic favourite) which is meant to be independent of self report.

It turns out that the BDI and HAMD scores were only weakly correlated - with a coefficient of just r=0.32. That's really not very good considering that, in theory, they both measure the same thing: 'depression'. Many people reported being considerably depressed when their clinicians rated them as fine.

But more interestingly, certain characteristics of the patients were correlated with their self-report/clinician-rating discrepancy. Specifically, patients with a lower IQ, who were more impulsive, and less conscientious, tended to self-report more severe depression.

Now, the uncharitable interpretation of these people is that they were just too sloppy to complete the form properly... the uncharitable interpretation of the psychiatrists is that it's their fault for underestimating depression in people less inclined to express themselves in 'the right way'. There's no way to know.

Either way, it's a serious problem because it shows that self-report and observer-report measures of depression aren't just poorly correlated, they're actually measuring different things for different people.

It could be even worse than it appears because the HAMD, although supposedly not a self-report measure, does in fact heavily rely on the patient's cooperation. So a 100% clinician-rated scale might be even further removed from self-report.

Kim EY, Hwang SS, Lee NY, Kim SH, Lee HJ, Kim YS, and Ahn YM (2012). Intelligence, temperament, and personality are related to over- or under-reporting of affective symptoms by patients with euthymic mood disorder. Journal of affective disorders PMID: 23270973

Flawed Statistics Make Almost Everyone's Brain "Abnormal"

A popular method for detecting abnormalities in the shape and size of individual brains is seriously flawed, and is almost guaranteed to find 'differences' even in normal people.

So say Italian neuroscientists Scarpazza and colleagues in an important new report: Very high false positive rates in single case Voxel Based Morphometry.

Voxel Based Morphometry (VBM) is a way of analyzing brain scans to detect structural differences. It's most commonly used to compare groups of brains to find average differences, but some neuroscientists have started using VBM to check for abnormalities in a single brain. Scarpazza et al list 34 pieces of research about that, including 13 since 2010.

So it would suck if there were a problem with individual VBM... but there is. This pic tells the tale:

The authors took 200 normal brains and compared each one of them in turn to a control group of 16 normal brains. Because all of them were healthy, the comparisons ought to show no significant differences.

The technique was set up so that, in theory, only 5% of the brains should have been wrongly labelled as containing an abnormality. But in fact, a full 93.5% of the normal brains gave at least one false positive.

So 5% is more like the rate of not being wrong. Oops.

The image shows that in some brain areas, almost 25% of the normal brains were branded as 'abnormal' just in that region alone - the hotter the colour, the higher the proportion of false 'hits'. The top row is for false reports of brain volume increases, while the bottom row is decreases; false 'increases' were more common.

So what's going wrong? It's not entirely clear and several factors are probably at play, but the authors say that the main issue is that VBM makes the assumption of statistical normality which doesn't in fact hold.

Either way, it's a serious problem, and Scarpazza et al point to one especially worrying implication: some people have proposed using single-subject VBM in a legal context, to reinforce insanity pleas by showing subtle 'brain abnormalities' not obvious to the naked eye. Yet if this paper's right, such evidence could be entirely meaningless, almost guaranteed to give a positive result.

P.S. Last time I posted about this kind of analysis flaw, the internet went crazy because they didn't understand it. So just to be clear, this is not a problem for clinical scans - the kind you'd get to check whether you have a brain tumour.

Scarpazza, C., Sartori, G., De Simone, M., and Mechelli, A. (2013). When the single matters more than the group: Very high false positive rates in single case Voxel Based Morphometry NeuroImage DOI: 10.1016/j.neuroimage.2012.12.045

The Perils of Sharing Brain Scans

A fascinating paper by neuroscientists Van Horn and Gazzaniga chronicles their pioneering, but not entirely successful, attempt to get researchers sharing their brain scans: Why share data? Lessons learned from the fMRIDC.

It all started in 1999 when, along with some colleagues, they decided that the time was right for data sharing in neuroimaging. They got some public funding, and tried to get various major neuroscience journals to require that anyone publishing an fMRI study should make their data available to the fMRI Data Consortium (fMRIDC).

By making it mandatory, they'd ensure that there was no selection bias. Requirements to post raw data were already common in other fields of science like genetics and crystallography. So, they thought, why can't it happen here?

However, it didn't go down very well:

Upon becoming aware of our efforts and goals, fMRI researchers angered by journal requirements to provide copies of the fMRI data from their published articles began a letter writing campaign seeking to muster opposition  an effort which was featured in the news and editorial sections of several influential journals.

Editorials and commentaries over fMRI data sharing were aired in the pages of Science, Nature Neuroscience etc. expressing concern over the data sharing requirement, over what possession of the data implied, human subject concerns, and, if databasing was to be conducted at all, how it should be conducted “properly”.

This was all before my time, sadly. It sounds like a grand old academic street-fight. No doubt those on the other side remember it differently from how it's presented here, though.

The reactions of our colleagues caught us somewhat off guard. We were honestly surprised by the  negative and hostile response when we had believed that creation of a data archive would be of benefit to the neuroimaging community. Perhaps, they had a point.

Maybe the field wasn't ready for fMRI data sharing? Perhaps, it was too early to start such a project? We struggled with how best to move forward or whether to move forward at all.

Anyway, they decided they would continue, on a more modest scale. fMRIDC ended up with data from about 100 fMRI studies by the time the funders pulled the plug in 2006, only a fraction (and perhaps an unrepresentative one) of the papers published, but still, it's something.

As I said, I missed out on this debate, but if I had been in the field 10 years ago, I suspect I'd have been on the extreme wing of the pro-sharing faction, the Montagnard to the founders' Girondism. My view is that no researcher owns their data. The only person who owns the results of a brain scan is the person whose brain it is.

If I could wave a magic wand, I'd put a chip in every MRI scanner that automatically uploaded all scans to a public database as soon as they appeared (with the subject's consent, and/or with personal information about the subject stripped out). I'd fix all the software such that every time someone ran an analysis, it was publicly logged. Total transparency is best for science, I believe, and it would also make scientists lives  easier, once they got over the initial shock.

Sadly that's not possible... yet... but data sharing is a noble cause and, as Van Horn and Gazzaniga point out, even if fMRIDC is dead, the idea lives on with many new initiatives emerging, hydra-like, in its place.

Van Horn JD, and Gazzaniga MS (2012). Why share data? Lessons learned from the fMRIDC. NeuroImage PMID: 23160115

Is fMRI About To Get Fifty Times Faster?

According to a paper just published, a new technique of functional MRI scanning (fMRI) could soon allow neuroscientists to measure brain activity far faster: Generalized iNverse imaging (GIN): Ultrafast fMRI with physiological noise correction

Authors Boyacioglu and Barth claim remarkable things for the technique:

We find that the spatial localization of activation for GIN is comparable to an EPI protocol and that maximum z-scores increase significantly... with a high temporal resolution of 50 milliseconds.

EPI, the current standard fMRI sequence, would have a temporal resolution of 2000 or 3000 milliseconds, so it's about 50 times faster.

Other super-fast fMRI methods already exist (e.g. this one I blogged about), but they've generally achieved speed only at a cost: they've had to either sacrifice spatial resolution to achieve that, or limited themselves to scanning only a small fraction of the brain, or have been more subject to random noise and hence less sensitive.

GIN, however, is said to cover the whole brain, with decent spatial resolution and signal-to-noise ratio. The data can be analyzed in exactly the same way as any other kind. So that's up to fifty times faster with no real drawbacks.

That would be truly revolutionary - as the major limitation of fMRI at the moment is that it's much slower than other methods of recording brain activity.

Check it out: this shows brain activation in response to simple visual stimuli, imaged with bog-standard EPI and GIN:

So this is a big deal... if it does work, I'm sure neuroscientists the world over will be lining up to buy Boyacioglu and Barth a GIN and tonic.

How does it work, and is it all it's cracked up to be? Well, I can't really say: the math is beyond me.

In essence, rather than scanning the brain in 3D, slice by slice (like this), GIN only scans one 2D slice, but then manages to reconstruct the rest of the brain in 3D from just that slice, using dark, forbidden magicks... I mean mathematics. The principle is called parallel imaging and it's been around for several years, but with image quality limitations that GIN claims to have overcome.

Perhaps my more technically-inclined readers will have more insightful comments.

Boyacioglu R, and Barth M (2012). Generalized iNverse imaging (GIN): Ultrafast fMRI with physiological noise correction. Magnetic Resonance in Medicine PMID: 23097342

When Replication Goes Bad

How to ensure that results in psychology (and other fields) are replicated has become a popular topic of discussion recently. There's no doubt that many results fail to replicate, and also, that people don't even try to replicate findings as much as they should.

Yet psychologist Gregory Francis warns that replication per se is not always a good thing: Publication bias and the failure of replication in experimental psychology

Among experimental psychologists, successful replication enhances belief in a finding, while a failure to replicate is often interpreted to mean that one of the experiments is flawed. This view is wrong.

Because experimental psychology uses statistics, empirical findings should appear with predictable probabilities. In a misguided effort to demonstrate successful replication of empirical findings and avoid failures to replicate, experimental psychologists sometimes report too many positive results.

Rather than strengthen confidence in an effect, too much successful replication actually indicates publication bias, which invalidates entire sets of experimental findings...

Even populations with strong effects should have some experiments that do not reject the null hypothesis. Such null findings should not be interpreted as failures to replicate, because if the experiments are run properly and reported fully, such nonsignificant
findings are an expected outcome of random sampling... If there are not enough null findings in a set of moderately powered experiments, the experiments were either not run properly or not fully reported. If experiments are not run properly or not reported fully, there is no reason to believe the reported effect is real.

Say you took a pack of playing cards and removed half the red cards. Your pack would now be 2/3rds black, so if you took a random sample of cards, say a poker hand of 5 cards, then you'd expect more blacks than reds (a significant 'effect' of color). But you'd still expect some reds, and some random hands would in fact be entirely red, just by chance. If someone claimed to have drawn 10 random hands and they'd all been mainly black, that would be implausible - "too good".

Francis's approach is a bit like Uri Simonsohn's method for detecting fraudulent data - they both work on the principle that "If it's too good to be true, it's probably false" - but they differ in their specifics, and I believe that we should not conflate fraud with publication bias... so let's not get carried away with the parallels.

Earlier this year, Francis wrote a critical letter about a paper published in PNAS purporting to show that wealthier Americans are less ethical. He argued that the paper's results were "unbelievable" - it reported on the results of seven separate experiments, all of which showed a small, but significant, effect in favour of the hypothesis.

Even if rich people really were meaner, Francis said, the chance of 7/7 experiments being positive is very low: just by chance, you'd expect some of them to show no difference (given that the size of the difference in those seven was low, with a lot of overlap between the groups). Francis suggested that the authors may have run more than seven experiments, and only published the positive ones; the authors denied this in their Letter.

Anyway, in the new paper, Francis expands on this approach in much more detail, drawing from this 2007 paper, and suggests a Bayesian approach that might help mitigate the problem.

Francis G (2012). Publication bias and the failure of replication in experimental psychology. Psychonomic Bulletin and Review PMID: 23055145

More on False Positive Neuroimaging

Back in June, I warned that the ever-increasing number of clever methods for analyzing brain imaging data could be a double-edged sword:

Recently, psychologists Joseph Simmons, Leif Nelson and Uri Simonsohn made waves when they published a provocative article called False-Positive Psychology - Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant.

It explained how there are so many possible ways to gather and analyze the results of a  simple psychology experiment that, even if there's nothing interesting really happening, it'll be possible to find some "significant" positive results purely by chance...

The problem's not just seen in psychology however, and I'm concerned that it's especially dangerous in modern neuroimaging research.

In a comment on that post, The Neurocritic pointed out that Michigan PhD student Joshua Carp had put forward the same argument in a conference presentation, several months previously.

Now Carp's published a paper on the topic: On the plurality of (methodological) worlds: estimating the analytic flexibility of fMRI experiments. It's free to access, so check it out.

Whereas I just talked the talk by listing lots of possible ways in which you could analyze a given set of data, Carp walked the walk, and actually did loads of analyses. He took a single dataset, the results of a simple experiment and looked at it in almost 7000 different ways. Each set of results was then thresholded to correct for multiple comparisons in 5 ways, for a grand total of 35,000 outputs.

The variants he considered ranged from how much smoothing to apply, to how to correct for head motion, and many more.

What happened? In a nutshell, the different options made a difference - and the variability was the largest in parts of the brain that were most activated (the "blobs" that lit up). In other words, analytic flexibility makes the most difference in the most interesting places. See the picture at the top.

The location of the maximum peak activation also varied. This is not unexpected, and not, in itself, that worrying - the great majority of the peaks clustered in a few small areas. However, it underlines that different options really can make a difference.

Carp concludes:

Nearly every voxel in the brain showed significant activation under at least one analysis pipeline. In other words, a sufficiently persistent researcher determined to find significant activation in virtually any brain region is quite likely to succeed...

If investigators apply several analysis pipelines to an experiment, and only report the analyses that support their hypotheses, then the prevalence of false positive results in the literature may far exceed the nominal rate. However, analytic flexibility only translates into elevated false positive rates when combined with selective analysis reporting. If researchers reported the results of all analysis pipelines used in their studies, then it would not be problematic.

To the author’s knowledge, there is no evidence that fMRI researchers actually engage in selective analysis reporting. But researchers in other fields do appear to pursue this strategy.

In my experience, fMRI researchers are actually fairly conservative in terms of using different analyses, and certainly I doubt anyone has ever run thousands of them just to get the result they want and I'd estimate that most published findings are not the result of more than a handful of 'attempts' at most.

However it's a serious concern that it could happen, and importantly it's getting ever-easier to do this, with the continuing increase in computer power making running an analysis quicker and cheaper than ever. As to what to do about it, Carp makes several suggestions, and here's one I made earlier...

Joshua Carp (2012). On the plurality of (methodological) worlds: estimating the analytic flexibility of fMRI experiments Front. Neurosci. DOI: 10.3389/fnins.2012.00149

Getting The Position Right For EEG

In science, it's often the most 'boring', easily overlooked factors that determine whether an experiment succeeds or fails.

A new paper reveals strong effects of body posture on brain electrical activity: Subject position affects EEG magnitudes. Just lying face-up as opposed to face-down can powerfully affect the signal measured using electroencephalography (EEG), according to Justin Rice and colleagues of New York.

Here's why: EEG uses electrodes, placed on the scalp, to measure the electrical potentials produced by brain firing.

The signal recorded depends, however, not just on the brain activity but also on the quality of the electrical conduction between the brain and the scalp: the signals have to travel through the fluids surrounding the brain, then the skull, and finally the skin, before they're detected.

EEG users sometimes think of brain-scalp conductivity as a fixed factor, that they can't control and don't have to worry about. However, Rice et al point out that the position of the brain shifts within the skull depending upon your posture.

This is because there's a bit of extra room in there, leaving space for the brain to "bounce around" a little within its fluid cavity. If you're lying on your back, the brain will lie closer to the back of the skull; if you're on your front, it'll be further forward, and so on. So the fluid layer between brain and skull will be corresponding thinner, or thicker.

In a healthy brain the change is only about 1 mm, but the fluid layer's only 3 mm at most, so that's a big change.

Others have recognized this problem before, but Rice et al's data are the clearest evidence yet that posture is a major factor. They showed that subjects lying on their back (supine) showed significantly stronger activity over the back of the brain - which makes sense, as it brings the brain closer to the electrodes. Lying face down (prone) made activity weaker and sitting was in between.

Interestingly - and worryingly - the effect was stronger depending upon the kind of activity being measured. For most kinds of brain activity it was about 40% higher but for gamma waves - the hottest thing in EEG right now - it was almost 80%.

So gamma band activity is especially sensitive to posture, and that raises the worrying possibility that even slight differences in head position between individuals could account for 'differences' in gamma power recorded, for example in studies comparing neurological patients and healthy controls; if the controls are sitting up straight while the patients are slouching back... the patients would seem to have more gamma.

Rice JK, Rorden C, Little JS, and Parra LC (2012). Subject position affects EEG magnitudes. NeuroImage PMID: 23006805

Militarization of Neuroscience?

US military tech hothouse DARPA have an exciting announcement:

Tag Team Threat-recognition Technology Incorporates Mind, Machine
DARPA links human brainwaves, improved sensors, cognitive algorithms to improve target detection...

In what is - to my knowledge - the first example of the direct militarization of neuroscience, DARPA have developed a system in which electrical responses in a human brain are an integral step.

A soldier watches a screen on which, via various cameras, possible battlefield "threats" are shown. The cameras are fancy, and fancy image-recognition algorithms prioritize images that resemble threats - stuff that looks a bit like a tank, an IEDs, etc. But that's just the set-up.

The neuroscience core is that rather than just having a guy watching this screen and pressing a button if he spots something, they have a guy wired up with EEG to record brain activity. The system registers a threat when a picture causes a P300 response.

Now, the P300 is an electrical wave triggered by stimuli that are somehow 'meaningful' to the individual person. If you ask someone to press a button whenever they see a red light, for example, and then show them various lights, red ones will elicit a P300.

Very clever. But it may be too clever for its own good.

We already have a system that can detect the P300. It's the brain. No, most of us don't think of it as in those terms - we think of it as "Oh!" or "WTF?" or "Button press time" - but that response is the P300 (or rather something that precedes it because the P300 takes 300 milliseconds to peak, but you can respond faster than that.)

So why the EEG?

You could program a computer to detect P300s in a guy's brain and set off an alarm. DARPA apparently have. But it would be easier and cheaper to just 'program' the guy's brain to detect the P300 and push an alarm button - by asking him to do that. The human brain is a supercomputer that's been in development for hundreds of millions of years and it's primary job is to detect threats and act on them as quickly as possibly. One day technology might be able to do better but I don't think we're there yet.

DARPA say:

In testing of the full CT2WS kit, the sensor and cognitive algorithms returned 810 false alarms per hour. When a human wearing the EEG cap was introduced, the number of false alarms dropped to only five per hour, out of a total of 2,304 target events per hour, and a 91% percent successful target recognition rate.

All that tells us is that having a human check the pics via EEG is better than having no human involved at all. That's fine, but would a human just checking the pics via a button, be even better? We're not told. Maybe DARPA ran those tests and it really does offer advantages, but off the top of my head I can't think of any, and it wouldn't be the first time that the allure of high-tech neuroscience has blinded smart people to the fact that there's an easier, less sexy solution.

Unless...

OK. This is going to make me sound like a conspiracy nut. But there's one scenario in which the P300 has a decided advantage: unlike a button press, it's involuntary. It would work even if the guy doesn't want to co-operate.

So suppose you've captured a terrorist and you want to know who his terrorist friends are or where they've put the bomb. But he's not talking and Samuel L Jackson is off sick. So you wire him up to this system and show him a bunch of pictures of all the possible suspects or targets on your database. His brain will respond with a P300 to the ones he recognizes.

That would probably work - sometimes - and the P300 is already being trialled in some legal contexts for just that purpose although it's not clear how reliable it is.

So it's just possible that this whole soldier-scanning-the-battlefield story is merely an elaborate front (and perhaps a useful source of crucial calibration data) for a device to allow the CIA to read minds. I warned you it would make me sound crazy. Quick! Pass the tinfoil hat...!

Cocktail-Party Neuroscience

"That's all very well, but what about the real world?"

This, or something to this effect, is a stock criticism of much of psychology and cognitive neuroscience. Studies of human behavior and brain function under carefully controlled laboratory conditions don't tell us much about everyday life, the argument goes.

It's a serious point. But a group of neuroscientists have now sought to dispel such worries in rather spectacular fashion. With the help of some nifty wireless headsets, Alan Gevins and colleagues of San Francisco took electroencephalography (EEG) out of the lab and organized an EEG party - allowing them to record brain electrical activity from 10 people as they chatted and drank vodka martinis. An electroencephalorgy one might say.

This is perhaps the only time in history that scientists have admitted, on record, to getting drunk with their research funding.

Pics or it didn't happen? They have pics:

And more:

The odd device held by the girl in blue is an alcohol breathalyser, which brings us onto the purpose of the study, which was to measure the effect of alcohol on brain activity.

The authors first measured the effect of alcohol on brain alpha, beta and theta band activity under standard lab conditions, and then checked to see if the results translated to the party. They did, surprisingly well in fact (although the whole thing relied on a multivariate model of the kind that make purists suspicious.)

Still, only 40% of the party data was deemed unusable due to electrical artifacts caused by participants speaking, swallowing, chewing and so forth, which is pretty good, and suggests that real-world EEG could be much more feasible than many neuroscientists would have predicted (given how annoying these sources of noise can be even under lab conditions I'd have guessed it would be more like 90%).

Now I'll make an admission: when I first read this paper, I was cynical. I felt sure it was some kind of advert for the authors' products, probably the nifty wireless EEG caps they used. "Oh very clever," I thought. "You run a wacky study, it goes viral, and you get free advertising. Well, it's worked on me, but I'm going to call you out on it."

However, the authors were one step ahead, because the paper assures readers that:

The authors are employed by the San Francisco Brain Research Institute and SAM Technology which are 100% supported by competing research grants from the U.S. Federal Government... The organization only performs research and offers no services or products. None of the authors perform consulting work. It is very unlikely that any corporation, investor, etc. would find it commercially worthwhile to buy or license the technologies that the authors have made to do their research.

OK then.

Gevins A, Chan CS, and Sam-Vargas L (2012). Towards measuring brain function on groups of people in the real world. PloS one, 7 (9) PMID: 22957099

When Data Filtering Introduces Bias (fMRI Edition)

A couple of months ago I blogged about a paper showing that 'filtering' of EEG data can create spurious effects.

Now, we read about another form of bias that filters can introduce, this time for fMRI: Filtering induces correlation in fMRI resting state data.

Australian neuroscientists Catherine Davey and colleagues consider temporal filtering of fMRI data in studies looking at correlation (brain functional connectivity).

Because both very high frequency and very slow changes in the fMRI signal are probably caused by artefacts, rather than interesting brain signals, it's common to use a filter to try and extract the medium-frequency changes that are of most interest (e.g. approximately 0.01 to 0.1 Hz).

However, while this filtering is very useful, Davey et al show that it can - ironically - create artefacts of its own: here's the data from one volunteer scanned during a simple task and then analyzed in 4 different ways:

Without filtering (A) there's a huge amount of 'connectivity' - too much to be realistic. This is why filtering is important.

But filtering, without correcting for the effects of the filter, actually makes things worse (B). It solves one problem but at the cost of creating another. The problem is those pesky autocorrelations. The authors say, however, that they've calculated a way to correct for filter-induced correlations (D) and that this gives more realistic results. They recommend that this should be used in future connectivity studies, but don't go into much detail regarding the question of what this means for the existing literature.

Perhaps data 'filtering' is a misleading term. It implies that all you're doing is removing the unwanted noise, leaving pristine, crystal clear data, a bit like a water filter. Mmm. What could go wrong? In fact mathematical 'filters' can put stuff into the data as well as take it out, so should we stop using that word and just call them what they are: modifications?

Davey CE, Grayden DB, Egan GF, and Johnston LA (2012). Filtering induces correlation in fMRI resting state data. NeuroImage PMID: 22939874

What Is Science?

The other day I was in a discussion about what "science" is. I've written about this before but this debate got me thinking about it again and I thought I'd set out what I think in more detail.

I wasn't sure how best to structure this so I'm going to start with my main claim, followed by a Q and A bit. The Q's are not intended to be straw-men or caricatures, they're questions I've asked myself in the course of thinking about this.

My Claim: "Science" is just the process of looking at the world and thinking about the evidence in an effort to understand it. It's not a special form of knowledge, scientists don't use a special 'scientific method' - scientists just look and think about things. They may use special equipment and techniques, but in essence it's no different to what we all do every day. As such it makes no sense to talk about the 'limits of science' or 'what science can't tell us', unless by that we mean the limits of human knowledge itself, because science just is knowledge.

Q: "But if science were just observation, everyone would be a scientist and it becomes meaningless." - No: for the same reason that not everyone is a poet, even though anyone can write a poem.

Science is observation informed by previous scientific findings i.e. it is expert observation. Anyone can, say, look at the stars but this doesn't make them an astronomer. Astronomy is in essence just looking at the stars and thinking about them - in a broad sense - but to contribute to astronomy, you first need to know the relevant background, which few except astronomers do.

Likewise, anyone can write a poem, but few of us can make a living out of it.

Q: "OK, but still, if science is just observation, then all forms of knowledge are science." - This is a tricky one, but the answer is crucial to understanding my point.

A few hundred years ago, the word "science" in English did indeed just mean "knowledge". However, more recently, it has come to mean a particular subset of knowledge: roughly, it today includes physics + chemistry + biology. Maybe some others.

We lump these three (or more) things together and call the lump "science". But this lump is more or less arbitrary. Physics + chemistry + biology don't share a special essence, which sets them apart from other kinds of knowledge (other 'sciences' in the older sense.)

So I'm not saying that all knowledge is "science" in the modern sense. The modern word "science" only includes a limited portion of knowledge. But I am saying that the rest of knowledge is essentially no different from science, because "science" is just an arbitrary subset of knowledge.

Here's a picture of what I mean:

Or here's an analogy. The color spectrum has infinite different shades. We conventionally divide it up into "red", "orange", "yellow" etc. and that's fine for most purposes. But there is no essential difference between "orange" and "yellow" and no clear dividing line: they are just collections of shades.

Science is a colour of knowledge. It's not a true kind.

Q: "But if the difference between science and knowledge is arbitrary, are you saying that The Scientific Method is the only way to knowledge?" Not at all. I don't think 'The Scientific Method' exists.

This follows from the fact that "science" is an arbitrary lump. Scientists are a diverse bunch and they use many different methods. Theoretical physicists, for example, use methods which are very close to those of mathematicians - who are not 'scientists' by most definitions. Zoologists use others, and you can get by in (most) of zoology without knowing any math at all. And so on.

In fact, there's almost as much difference between branches of "the same science" as there is between sciences. Just like "science", "biology" is a lump of diverse things, although not quite as arbitrary a lump.

Outside science, people use all kinds of methods as well. Historians have their set of methods, economists have others, all tailored to the particular demands of the case. That's exactly how it should be - all knowledge comes from observation and, to observe different things, you need different methods.

There are many different kinds of facts, but a fact is a fact, whether it's a scientific fact, a historical fact, or just an everyday fact. "Shakespear wrote Hamlet" is just as true as "The earth orbits the sun" is just as true as "It's raining" (if it is, in fact, raining.) The facts of history are just as true as the facts of biology.

But there is a grain of truth in the question here: I am saying that observation is the only way to knowledge.

Q: "That's very simplistic. Are you saying that we can only know what we can see and measure?" - No because I'm using "observe" in the broadest sense here, to include things like noticing, sensing, feeling, seeing, hearing, being told that, reading about...

You observe that it's raining: maybe you look out the window, maybe you hear the rain on the roof, maybe someone comes in from outside dripping wet. You observe that you feel hungry. You observe that Obama is president (even if you've never actually seen him) by watching the news. Etc.

"Observe" is in many cases an awkward word and I'm not saying we should use it in all those cases. My point is that these are all ways of finding stuff out about the world. In that broad sense, I think all knowledge comes from observation - although maybe indirectly, through thinking about observations; thoughtless observation is almost as unhelpful as purely abstract speculation.

Of course, it's easier said than done: very often it's not clear what we've observed (is it real or some kind of trick, mistake, illusion?), what it means, whether it matters, or even what the question is. This is why there's always room for debate, controversy and doubt, at least at first, before all the observations have been made.

Q: "But what you're saying is merely trite and obvious: 'science is based on observation' 'there is no one single scientific method', well, duh!" - From a certain perspective, they are obvious, but if you think so, then you ought to agree with everything else I've written here. They're all connected, you can't take some and leave the rest.

Beyond Self-Report

If you want to learn about someone, should you ask them?

Two bits of research published recently cast doubt on the validity of self-report as a tool in psychology and psychiatry. The first found that teens who reported that they suffered from bullying experienced more mild 'psychotic-like' symptoms. That correlation would be consistent with the idea that these symptoms arise as a response to stress.

However - the same study found that there was absolutely no correlation between peer ratings of whether someone was bullied, and their psychotic symptoms. Only self-report was associated.

The second study looked at whether a crisis intervention program - intensive mental health care - helped people who'd recently attempted suicide. The results showed that, compared to a control condition, suicidal patients who got crisis care self-reported fewer subsequent suicide attempts.

But a trawl of hospital records painted exactly the opposite picture - the intervention group were more likely to end up in hospital with a second attempt.

Remarkably few studies in psychology and psychiatry compare self-report to other measures of behaviour. This is because self-report is typically a lot easier. If these papers are anything to go by, however, this is a serious limitation. Self-report can be radically at odds with other measures of behaviour.

Which raises the question - who's right? Are the self-reports right, when they clash with other sources? I don't think there's an easy answer. In the bullying case, it might be that self-report is more accurate, because the peers doing the peer-report are the bullies. In the suicide study, maybe the self-report was more accurate, because the patients knew about attempts that never made it to hospital.

But on the other hand you could argue exactly the opposite. Maybe the self-reports of bullying just reflected whether the kids thought their classmates liked them. Maybe the patients were ashamed to admit that they'd reattempted suicide even though they'd got all this special crisis care. It's hard to tell.

One thing's clear though: self-report is not the whole story.

Gromann PM, Goossens FA, Olthof T, Pronk J, and Krabbendam L (2012). Self-perception but not peer reputation of bullying victimization is associated with non-clinical psychotic experiences in adolescents. Psychological medicine, 1-7 PMID: 22895003 

Morthorst B, Krogh J, Erlangsen A, Alberdi F, and Nordentoft M (2012). Effect of assertive outreach after suicide attempt in the AID (assertive intervention for deliberate self harm) trial: randomised controlled trial. BMJ (Clinical research ed.), 345 PMID: 22915730

Replication Alone Is Not Enough

Psychology has lately been hit by high-profile fraud scandals, and broader concerns over questionable research practices. Now the Society for Personality and Social Psychology (SPSP) has released a statement on "Responsible Conduct", and a task force has produced a report.

This is a start, and the SPSP is to be commended for facing up these problems (which affect many other fields) relatively early. However, neither of their documents contains much meat in my view.

Point One on the task force report is that "Replication is the key to building our science" and they suggest a "web site for depositing replications and failures to replicate" - but don't mention that various enterprising researchers have already made one. Nor do they tip their hats to the Open Science Initiative addressing just this issue. This makes me worried that they're planning to reinvent the wheel.

More fundamentally I disagree that replication is key to psychology or any field. Our goal should be replicability. Failure to replicate findings is a symptom of problems with those original findings, rather than being a problem in and of itself. Good results replicate; we want better results to be published.

In other words, we should strike at the root cause of invalid research, namely, the perverse incentives towards publishing as many eye-catching positive results with p values below 0.05 as possible by any means necessary. P-value fishing, selective reporting, post-hoc "prior hypotheses" and other questionable practices are a large part of what make unreplicable results.

We should encourage replication, but it's no panacea.

An overemphasis on replication, without addressing the incentives, could actually harm science. It could lead to scientists spending all their time worrying about the political drama of who's replicating who and why, and which questionable practices they can use to replicate their friends' data - rather than actually doing science.

This is why we shouldn't be satisfied with any reform effort that puts replication before replicability. If you can fudge a result, you can fudge the data a replication. How to fight questionable practices is another question but I've proposed reforms that I think would work, namely pre-registration of hypotheses, methods, and statistical analyses. Others have their own ideas.

A lesson from clinical medicine here. Clinical trials of new drugs adopted pre-registration, but only after they tried replication and it didn't work. Pharmaceutical regulators have long required multiple demonstrations of drug efficacy. One trial was not enough. Sounds good - but the problem was that drug companies just did lots of trials and analyses, picked the positive ones, and used them.

So in summary: replication is important, and we don't do enough of it, but replication alone is not enough to fix psychology.

Neuroscience: Solving The Hard-On Problem

Take a good look at this, fellas:

It might not look like much, but this is science's very first glimpse of something rather close to the hearts of most men.

These images show nerve activation in the spinal cord during sexual arousal. Using functional magnetic resonance imaging (fMRI), best known as a way of recording brain activity, applied to the spine, Canadian researchers Kozyrev et al were able to record the changes associated with, well, stimulation. Here's the paper: Neural correlates of sexual arousal in the spinal cords of able-bodied men.

The methods sound like a real riot. Ten guys were scanned while being exposed to two different kinds of stimulation. First, 'audiovisual'

Films, depicting heterosexual couples engaged in sex play, vaginal sexual intercourse, and oral sex, were presented in two 5-min blocks, separated by a baseline condition (blank screen) for 3 min... The selected films were chosen by the investigators and were all very highly rated (on a scale of 1 to 10) by independent viewers. Only films that received a minimum score of 8 out of 10 were shown to participants.

They don't say who these independent grumble-rating viewers were, but I'm, er, told that some porn sites these days have user ratings next to each clip. So it's possible that they used those and that the raters were, quite literally, just some bunch of wankers.

As for the manual stimulation...

All participants manually performed stimulation of the penis. Participants received written instructions superimposed onto a rear-projection screen, indicating whether to rest or to perform penile self-stimulation... there were two equal blocks of 1.5 min of stimulation that were separated by a 1.5-min block of rest... a shorter duration was selected for this part of the study in order to avoid overstimulation of the penis and/or ejaculation.

Good thinking. That could have been messy.

The results of the study largely confirmed previous work, but it's an important step forward because until now, no-one had measured arousal-related spinal activation directly in healthy humans; the evidence came from recordings in animals and spinal injury patients.

Clearly there's more work to be done here. The images are rather crude. That grey oval is meant to look like this:

Still, it's early days. Spinal cord fMRI is not a new technique, having first been performed in 1999, but it's not received nearly as much research interest as the brain variety.

Kozyrev N, Figley CR, Alexander MS, Richards JS, Bosma RL, & Stroman PW (2012). Neural correlates of sexual arousal in the spinal cords of able-bodied men: a spinal FMRI investigation. Journal of sex & marital therapy, 38 (5), 418-35 PMID: 22900624

Questionnaire Extremism and National Character

"Personality differences" between people from different countries may just be a reflection of cultural differences in the use of 'extreme' language to describe people.

That's according to a very important paper just out from an international team led by Estonia's René Mõttus.

There's a write up of the study here. In a nutshell, they took 3,000 people from 22 places and asked them to rate the personality of 30 fictional people based on brief descriptions (which were the same, but translated into the local language). Ratings were on a 1 to 5 scale.

It turned out that some populations handed out more of the extreme 1 or 5 responses. Hong Kong, South Korea and Germany tended to give middle of the road 2, 3 and 4 ratings, while Poland, Burkina Faso and people from Changchun in China were much more fond of 1s and 5s.

The characters they were rating were the same in all cases, remember.

Crucially, when the participants rated themselves on the same personality traits, they tended to follow the same pattern. Koreans rated themselves to have more moderate personality traits, compared to Burkinabés who described themselves in stronger tones.

Whether this is a cultural difference or a linguistic one is perhaps debatable; it might be a sign that it is not easy to translate English-language personality words into certain languages without changing how 'strong' they sound. However, either way, it's a serious problem for psychologists interested in cross-cultural studies.

I've long suspected that something like this might lie behind the very large differences in reported rates of mental illness across countries. Studies have found that about 3 times as many people in the USA report symptoms of mental illness compared to people in Spain, yet the suicide rate is almost the same, which is odd because mental illness is strongly associated with suicide.

One explanation would be that some cultures are more likely to report 'higher than normal' levels of distress, anxiety - a bit like how some make more extreme judgements of personality.

So it would be very interesting to check this by comparing the results of this paper to the international mental illness studies. Unfortunately, the countries sampled don't overlap enough to do this yet (as far as I can see).

Mõttus R, et al (2012). The Effect of Response Style on Self-Reported Conscientiousness Across 20 Countries. Personality and Social Psychology Bulletin PMID: 22745332

On Twitter, It's Beer Before Liquor

People tweet about beer in the evenings, especially on Fridays, according to a not-very-surprising-but-still-fun little report in the journal Epidemiology: Using Twitter to Measure Behavior Patterns

The study used timeu.se, a free searchable database of millions of Tweets. The site grew out of an excellent bit of research you might remember from last year that examined how average mood varies over the course of the day and year.

People tweet about and presumably drink beer (and wine) most in the evenings, especially on Fridays and Saturdays. Smoking-related tweets however were mostly flat, although there did seem to be a small peak in the mornings.

So I ran a few searches of my own, and I noticed that different drinks peak at different times. "Beer" is most popular in the early evening, closely followed by "Wine", but "Vodka" doesn't max out until midnight.

Looks like people are not following the old "beer before liquor, never sicker" rule.

Also, while wine gets the most tweets during the week, beer and vodka really take off at the weekend.

 Cunningham A. John (2012). Using Twitter to Measure Behavior Patterns Epidemiology DOI: 10.1097/EDE.0b013e3182625e5d

Brains In Motion Are Bad For Neuroscience

A new paper in Human Brain Mapping reports on: Functional magnetic resonance imaging movers and shakers: Does subject-movement cause sampling bias?

Head movement is a well known problem that can badly impact the quality of neuroimaging data, introducing spurious signals and obscuring real ones. It's an issue for all brain scanning research but according to Wylie and colleagues, authors of this paper, it's especially serious for studies comparing disease patients to healthy controls.

The authors got 34 people with multiple sclerosis (MS) to perform some simple cognitive tasks during fMRI scanning. They found that the harder the task was, the more the patients moved around during the scan; and in those with more severe MS, the correlation between difficulty and motion was even stronger. In healthy people, harder tasks only caused slightly more motion.

And the more people moved, the less brain activity was recorded, probably because movement degraded the data quality.

fMRI data associated with severe head movement is frequently discarded. In discarding these data, it is often assumed that head-movement is a source of random error, and that data can be discarded from subjects with severe movement without biasing the sample. We tested this assumption by examining whether head movement was related to task difficulty and cognitive status among persons with multiple sclerosis (MS).

[In the MS patients] there was a linear increase in movement as task difficulty increased that was larger among subjects with lower cognitive ability. Analyses of the signal-to-noise ratio (SNR) confirmed that increases in movement degraded data quality. Similar, though far smaller, effects were found in healthy control subjects. Therefore, discarding data with severe movement artifact may bias multiple sclerosis samples such that only those with less-severe cognitive impairment are included in the analyses. However, even if such data are not discarded outright, subjects who move more will contribute less to the group-level results because of degraded SNR.

Oh dear. fMRI researchers use two main ways to deal with motion - correction, and rejection. Either you try to take account of movement and analyze the data, or you just throw out the results from people who move a lot. Most people use a combined approach, chucking out the really heavy movers and then using correction on the rest.

This paper however shows that both techniques have problems. Despite motion correction, data from heavy movers has a lower signal to noise ratio, so if you include them, they will "dilute" your sample. However, if you chuck them, that'll also introduce bias, because heavy movement is not random - people with more severe MS move more, so by excluding heavy movers, you'll be excluding severe cases.

Informally, every neuroimaging researcher knows that some people move more than others. Learning to spot likely "movers" and avoid wasting money on scanning them is a fine art. In my experience just about every "patient" population move, on average, more than healthy controls, and children and the elderly move more than young adults.

I'm not sure there's an ideal solution but perhaps the best approach is to run all analyses (at least) twice, once including everyone, regardless of movement, and then again, with strict movement exclusion criteria. Results consistent across both analyses are probably solid.

Wylie GR, Genova H, Deluca J, Chiaravalloti N, and Sumowski JF (2012). Functional magnetic resonance imaging movers and shakers: Does subject-movement cause sampling bias? Human brain mapping PMID: 22847906