Suppose you're a company or activist group. You commission a survey of 2,000 people, and ask them some questions vaguely relating to your product or cause. You pick the most interesting results, write them up into a publication-ready press release, and send it to journalists. There's a good chance that your press release will appear, with minor alterations, as a news story in the British media. Like these articles (BBC, Daily Telegraph), which bear a striking similarity to this press release.Which is good news for you. You get your name in the papers, and it doesn't even look like advertising. Journalists get column inches for very little work, and the pollsters who conducted the survey get publicity too (and your money). Everyone wins, except the public, who end up bombarded with usually meaningless statistics in the guise of "research".
The survey doesn't have to be of 2,000 people, but that's the norm. This is because this is the size of the samples used by YouGov PLC, who are responsible for (at least it seems to me) the majority of these things.
YouGov polls are everywhere. I'd always assumed that they were telephone surveys of a random sample of the British population. I assumed that because I thought that they were meant to be representative. How naïve.
In fact YouGov polls work like this: you sign up as a panelist, online, which takes two minutes. You then occasionally get e-mails inviting you to do surveys. If you do one, you get 50p credit. When you've got £50 they send you a cheque. It's a great way of making cash online, according to websites about making cash online. Sign up here to get in on the action.
So, the participants in all YouGov polls are not random people but are both self-selected and financially motivated. Many of them will be just doing it for the cash, in which case they will be trying to answer the survey as quickly as possible.
Worst, the panelists are not representative of the British population - they consist of people who use the internet, have heard about YouGov, and chose to participate. YouGov say they have 200,000 users, out of 60 million British people.
Every day, YouGov sends a survey to a certain sample of their users which collects 2,000 responses. They call this the "Omnibus" poll. It costs £500 to commission one, according to the leaflet. That's chump change in advertising terms; I don't know how much it would cost to run an ad in one or more newspapers, but it would be much more. With a YouGov poll you might even get onto bbc.co.uk, which doesn't do paid advertising at all. You can see why it's so popular.
YouGov defend their methodology against criticisms. Their main argument is that their approach has a track record of being accurate in predicting the outcome of British elections. But political polling is unique. Politics is one of the few things that most people have strong opinions about. And elections are just big polls, after all. Pollsters can learn through trial-and-error the best ways of weighting their results to achieve accurate predictions.
So the fact that YouGov are good at predicting elections doesn't mean that their polls are any good at probing the nation's drinking habits, attitudes to the mentally ill, favorite vegetables, or whatever else. They could be totally wrong. Or they could be perfect. We don't know. It doesn't matter to the people who commission these surveys, of course, because it's publicity either way. It should matter to journalists, but it doesn't seem to.
Bottom line: if you want cheap media exposure, call YouGov. If you want serious news, don't. And if you want to know how journalism got into this sorry state, read Bad Science and Flat Earth News. Really. Bloggers like me are not going to shut about those books until everyone's read them at least five times.
[BPSDB]
6 comments:
Sounds just like amazon turk
How naive you were indeed, and still are. A bit stupid, too. Don't you realise that, if you invite 2,000 people out of 200,000 to take part in a survey, that is still random? It takes years to reach a reward of 50 pounds, so the financial interest is minimal. The reason why YouGov always gets the right results is that people are less inhibited to tell their true opinion to a computer than to a personal caller.
A recent YouGov poll found you're wrong, Karl.
The panel of 200,000 are not a random sample of the British population, at all. And they don't pick the subsample of 2,000 entirely randomly, they send the survey to more than that and only some of those complete it; they stop when they reach 2,000...
And they don't send it to a random sample of the panel, they send it to a weighted random sample, weighted for the purposes of political polling, which may not apply to other questions. And if the financial interest is minimal, why do so many "how to make money online" sites recommend it?
Why is being "less inhibited" necessarily a good thing since it could just equate to "unthinking" (again, maybe good for political polling, not for other stuff?). We have no way of knowing whether it is accurate for anything except election voting intentions.
And why do you love YouGov?
You are right about my wrong use of the word "random" but you know what I meant. If YouGov's methodology of selection and weighting works so well with elections, why should it not work for the other stuff? And yes, I do think that less inhibition in opinion polls will always deliver truer results. For an extreme example, an "unthinking" racist racist is more likely to own up to his true opinions in an online survey that guarantees his anonymity than in a face-to-face or telephone interview where he might be ashamed of them. Why should it be different for non-political matters?
I think the YouGov reply and my response addresses most of those points but I just wanted to add something here:
"And yes, I do think that less inhibition in opinion polls will always deliver truer results. For an extreme example, an "unthinking" racist racist is more likely to own up to his true opinions in an online survey that guarantees his anonymity than in a face-to-face or telephone interview where he might be ashamed of them. Why should it be different for non-political matters?"
The difference is that political polls ask about opinions, for which a quick and honest response is probably the most "true". But non-political polls often ask about facts e.g. "How often do you do X?" or about factual beliefs e.g. "Do you know how to do Y?". Uninhibited answers as not necessarily better here, they might be worse if people just rattle off the answers without really thinking about it.
Although a telephone survey would probably be just as bad.
Post a Comment