Stop Reporting Surveys Wrong

There’s a common mistake in the way findings of surveys are reported. It annoys me every time I see it, and when I saw it again yesterday it annoyed me enough to write on my blog about it. Here’s a screenshot of a tweet promoting the article. Can you spot the mistake?

Pretty awful, right?

So, the survey asked people which groups they thought faced a lot of discrimination in the United States today. 44% of white evangelicals said Muslims did, and 57% of white evangelicals said Christians did. So more white evangelicals think Christians face a lot of discrimination than think Muslims do. But that’s not what the headline says.

The headline says that white evangelicals think Christians face more discrimination than Muslims. And it follows from the survey’s findings that at least 13% of them do think this, since at least 13% think Christians face a lot and don’t think Muslims do. But it’s still open that 87% think that Muslims face more discrimination. I guess it probably isn't as high 87%, but we just don't know what the figure is. If a survey just finds that at least 13% of people in a group think something, it’s misleading to report the survey as finding that the group as a whole thinks that thing. And if that thing is both false and a dangerous and foolish thing to think, as in this case, it’s especially important not to misleadingly report the group as thinking it.

Now, you might say that from the results of the survey it’s more or less certain that most white evangelicals think Christians face more discrimination than Muslims. I don’t think it is. But even if it is, the reader can be the judge of that. Report what the survey says, and let the reader draw their own conclusions. Or argue for a conclusion. But don’t just misreport the findings of the survey.

It's harsh, but does it work?

West Brom are a popular club. They play entertaining football, they don’t expect to spend more than a few seasons at a stretch in the top flight, and when they play against local rivals Aston Villa I’d rather cheer with Adrian Chiles than David Cameron. ‘The Baggies’ is an endearing nickname, too.

In spite of these four good reasons to the contrary, I’m going to be booing West Brom for the rest of the season. I hope they go down ignominiously, and if they don’t get their act together and go back to being a club Chiles can be proud of, then I hope they spend a good few seasons in the doldrums where they can think about what they’ve done. And what exactly have they done? They’ve sacked their charming manager from his first big job after he’d secured them automatic promotion last season and had been meeting if not exceeding expectations in the notoriously difficult first season back in the Premiership. Then they replaced him, not with someone like Mourinho or Hiddink who really would have been a step up, but with the latest journeyman to find himself out of work. Sound familiar? It should do. Those paragons of realism Newcastle United did the same thing in December.

Now, the most successful English club in recent decades has been Manchester United, and they’ve stuck with the same manager over that time. Even the dimmest misinterpreter of statistics will realise that there’s a chance the managerial stability was caused by the success and not the other way round. So if you want to know whether clubs benefit from sacking managers who seem to be losing their stuff, it’s not as simple as comparing stability with success. We should also be wary of being overly impressed by bounces in performance after a sacking, partly because short-term results have a lot of random noise in them, and partly because it doesn’t take great results to do better than a series of losses, and usually managers are sacked after a series of losses.

I’d have a lot more sympathy for boards sacking managers so frequently if there was evidence that it was good business, but it’s hard know what to measure. I’m not sure how to do it, but I’d be surprised if Steven Levitt couldn’t work it out. It’s right up his street. It’s as if he read somewhere that Galileo said “measure what is measurable, and make measurable what isn’t” and thought “you know, that’s not such a bad idea!”. Freakonomics isn’t the best popular economics book I’ve read; Tim Harford’s The Undercover Economist is. But finding out whether sacking Di Matteo was a good idea is definitely Levitt’s department.