Sunday, 22 August 2010

PR Reviewed Research

Suppose you've done some research, but unfortunately, it's crap.

Maybe your methods are flawed. Or your data don't really support the conclusions you want to draw from them.

You seem to be out of options. You could release the research, but then people would criticize it, or you could keep quiet about it, but then you've wasted all the time and money you spent on it. Neither is very attractive.

But there's a third option. Publicize the conclusions of your work, along with the best cherry-picked results, before you actually release the full report. Write a press release which "for reasons of space" only discusses the sexy stuff. You could even make it out to be a "leak", if you were feeling really devious.

Everyone will start talking about what you've said, despite the fact that without the full data, it's just your claims that you might as well have pulled out of thin air. Yet no-one can criticize it because no-one knows what your methods were. Leave it a few weeks, and then when you eventually do release the details, no-one will care any more - but the message has got out there.

*

On an unrelated note, a British management consultancy firm have done some research concluding that British local government employees work less efficiently than their counterparts in business, due to poor management. Sounds like a (very very big and lucrative) job for a management consultant!

Unfortunately, the details of the research aren't available yet. Their website tell us that
The research into public sector productivity will be available as a down load from this site when the report is released at the end of August.
But the conclusions are available right now, and are all over the media, including even the BBC whose remit must have expanded to cover advertising while I wasn't looking.
Junior staff in local authorities were, on average, productive only 32% of the time during working hours, said [the] management consultancy... It said this compared with an average of 44% in the private sector.
Is this true? We have absolutely no way of knowing because all we're told about the methodology was that it involved
1,855 surveys of managers and supervisors (173 from local government officers), 376 day-long observations, comprised of a minute by minute categorisation of how the manager in question spent his time, of which 36 were from local government.
Sounds like it could be pretty solid. Or it could be complete bollocks. The devil is in the details as it always is with research: what were the survey questions? Were the samples representative? What was the compliance rate? Were the people who did the minute-to-minute categorization of manager's time blinded to whether the manager was public or private sector?

No doubt we'll be informed as to all this in about two weeks, by which time no-one will care - but the message has got out there.

Fantastic.

6 comments:

veri said...

lol! Absolute bullocks, due to no money, no golf and everything else in the povo public sector. Knowing that isn't going to put bread and butter on my table so I don't think anyone in the whole wide world cares except for those corporate lawyers flashing badonkadoks at govt tenders/decisions on behalf of private firms.

Privatisation is an UGLY ongoing war.

Bravo Neuroskeptic! In solidarity :)

Anonymous said...

Alternately, 56% or 68% of work time is spent slacking off?

Bernard Carroll said...

A variant on this theme is Post-Publication PR. Here’s how it works. You publish a crappy little report in a minor journal that few people read, then you arrange press interviews, issue publicity releases, and write academic review articles talking up your nonsignificant findings. A dash of hyperbole can go a long way, especially when you are raising capital for your startup company. Hardly anyone will go take a critical look at the original report.

It also helps to give your cronies stock options and then to have them talk up your drug without disclosing their competing financial interest.

There is no shortage of recent examples of such shenanigans.

Anonymous said...

I know nothing of this particular piece of management research, but I think in general terms you are probably overstating the responsibility of the researchers in hyping their work.

I used to work in a PR office for a publisher of scientific journals, mostly chemistry-related. We would go through the upcoming articles, identify research "of interest to the public", contact the researchers, and then write up press releases and coordinate all the PR. The journals were all peer-reviewed, and we weren't qualified to judge the scientific merit (many of us didn't have even a bachelor's degree in a science), so we didn't even try. The key question was whether the conclusions would be interesting to the public.

The researchers were generally quite surprised to be drawn into the whole sordid world of PR, and pretty naive as to how it worked. The real beneficiary was us, the publisher, not the researchers. Universities and research institutes have PR offices which do much the same thing.

If anything, my guess would be that researchers whose work gets hyped in the popular press are generally displeased with the whole experience. At best, their careful caveats get lost. At worst, their conclusions get horribly distorted.

There are many cases of science bloggers debunking some ridiculous science "news" item by simply calling up the researchers and asking whether the news articles accurately representing their research. Frequently the answer is no.

Neuroskeptic said...

Anonymous: You're quite right, when it comes to academic research, it is usually (though not always) someone other than the researcher who does the hyping.

But in my example, it's pretty clear that the people who did the work were the ones who sent round the press release.

Gwailo said...

Reminds me of a study done back in the old days of the Soviet Union, which found that 57.2% of all statistics in Uzbekistan were falsified.