Reviewing Non-Fiction Is Hard

Sometimes I read non-fiction books, and really enjoy them. “What an awesome book,” I’ll think. But it’s actually quite hard to tell if non-fiction books are any good or not. At least, it’s hard to tell just by reading them. I guess you could read a review. But someone has to write the reviews.

The reason it’s hard is that you’ll usually be in one of two situations. Either you’ll be an expert in the topic the book is about, or you won’t be. Suppose you’re not, and so a lot of the stuff in the book is new to you. You don’t know if the book is any good or not, because you don’t know whether the stuff in the book is right or not. You can try factchecking it, but even if you can track down the sources they’ll often be buried in difficult academic writing of a sort you’re not really competent to understand. And if the book you’re reading is any good, a lot of what it’s telling you won’t be checkable facts, but rather a kind of expert insight and analysis that you wouldn’t be able to reconstruct yourself. And of course some books contain original research, in which case they can’t really be checked because in a way they are the source. These three all blur into each other, but they all make it very hard for a non-expert to tell if a non-fiction book is any good just by reading it. (And if we’re being strict about “just by reading it”, you’re not allowed to factcheck it anyway! But let’s not be strict. It's hard in any case.)

A good example of this danger is Daniel Kahneman’s Thinking, Fast and Slow. Kahneman is one of the world’s top psychologists, he did pioneering work on cognitive biases with Amos Tversky, and he won the economics Nobel "for having integrated insights from psychological research into economic science, especially concerning human judgment and decision-making under uncertainty". The book is about cognitive biases and decision-making under uncertainty, and the title refers to two styles of thought, one automatic and not very conscious and the other fairly carefully thought through and more conscious. It tells you about lots of cool little findings from psychological research, like how people are more likely to believe something they read if it’s written more legibly (pp62-3), and it integrates the anecdotes into a general narrative about the different ways people think.

When that book came out, it was very well received by people who weren’t already experts on the topic. I vaguely remember experts being more divided on it, but I’m not an expert and I thought it was awesome. I was entertained by the anecdotes, and I really felt like I was learning something. I told my friends the anecdotes as if we could be confident that they were true, and I recommended the book to them. But since it came out, psychology has had a bit of an existential crisis based on the fact that lots of its little findings don’t replicate. A lot of effects people found may well have been flukes that only seemed representative of how people behave because when psychologists have done experiments and not found anything cool they haven’t told anyone about them. Or they have but nobody has listened, which makes them less likely to bother telling people next time. Kahneman himself is very concerned about the whole thing, and he thinks he was a bit too credulous about some of the stuff in the book.

Now, I don’t want to set Kahneman up as some kind of fall-guy here. It may still be a good book, and the issues with some of the anecdotes may be kind of minor. He’s still a great psychologist and communicator, he was writing in good faith, and his willingness to publicly address problems with his own work is impressive and an example to his colleagues. The point is that I wasn’t competent to judge how good his book was. And to be honest, I still couldn’t tell you.

So, it’s hard to tell if a non-fiction book is good if you’re not already very familiar with the subject. Well, duh! But what if you are an expert? What if you could have written the book yourself? In that case you have a different problem: The Curse Of Knowledge. In his book The Sense of Style, Steven Pinker argues that the reason people often write badly is they struggle to imagine what it’s like not to know things that they in fact do know. You know what you want to say, but your reader has to try to work it out from what you’ve written, and it’s hard to put yourself into their mind and work out whether what you’ve written would still be clear. And if you’re trying to explain something you already understand but they don’t, you have to put yourself into their mind and work out whether they can understand the thing you’re trying to explain on the basis of what you’ve written. That’s hard too. Now, imagine you’re an expert reading a book by another expert on the same thing, and you’re trying to work out whether a non-expert will be able to understand the thing the book is about on the basis of what the book says. It’s not easy to do.

So, who should be reviewing books, if both experts and novices face systematic obstacles? Three suggestions come to mind.

• Someone who is neither an expert nor a novice. What you want is someone who doesn’t already know what the book says, and so doesn’t have the curse of knowledge, but who’s competent enough in the sort of thing the book is about to be able to check if it’s right, once they’ve been told the things the book says. In general it’s often easier to check an answer to a question than find the answer. While I wasn’t competent to check if Kahneman’s book was any good, maybe a psychologist in a different field could have done it.
• A great teacher. The curse of knowledge essentially arises because a certain kind of imaginative exercise is difficult. But some people seem to be quite good at it. Being good at it is part of the skillset of a great teacher: they need to be able to get into the minds of the students and tell whether what they’re saying would communicate the material to someone who wasn’t already familiar with it.
• A novice and an expert working together. The two problems are pretty separate, so in theory the expert ought to be able to read the book to check that what it says is sound, while the novice reads it to see if they feel like they’re learning something. And after they’ve both read it, the novice and the expert can talk to each other so the expert can check that the novice really did learn the things they felt like they were learning.

My favourite one is the last one. While an expert in an adjacent field might be able to do a decent job of factchecking, they won’t do as good a job as an expert, and it’ll be harder for them. Their transferable research skills will also mean they still have to do a bit of difficult imagining to get into the minds of the intended audience. A great teacher might be able to do the job, but we don’t even have enough great teachers to fill all the teaching jobs, let alone all the reviewing jobs as well. This leaves the last option. Unlike great teachers, experts are ten a penny, and novices are twenty a penny. Of course, you do need two people. But it’s still my favourite option, and it’s kind of odd that it practically never happens.

Wrong, wrong, wrong

This week I read Simon Baron-Cohen’s book Zero Degrees of Empathy. I didn’t like it. He may well be best known among laypeople like me for his work on differences in the distributions of psychological traits among people of different sexes. That comes up in this book, but mostly incidentally, and in any case my problems with it aren’t anything to do with that.

Zero Degrees of Empathy is a book about empathy. Baron-Cohen (yes, he is Borat’s cousin) thinks that the common characteristic of three personality disorders and two types of autism is that people with the conditions have little or no empathy. He also thinks that we should stop explaining human cruelty in terms of evil, which is a bit of a non-explanation, and start explaining it in terms of a temporary or permanent deficit of empathy on the part of the perpetrators. I’m not sure how common it is these days to give a heavyweight role to evil in the explanation of cruelty, but if studying empathy helps us understand cruelty better, then empathy’s worth studying. No problems there.

My problem with the book is that it seems to be riddled with errors. Here are some of them:

• On page 14 there is a graph showing that he’s found people’s empathy levels to be normally distributed. He says it’s a spectrum, but divides it into seven evenly spaced levels, from level 0 (no empathy) to level 6 (Desmond Tutu). The graph shows the mean/mode/median to be at level 3. Then on pages 17-20 he describes what each level is like. In the descriptions, people at level 3 have to ‘pretend to be normal’, and may realize that they ‘just don’t understand jokes that everyone else does’. Level 4 is ‘low-average’, and level 5 is ‘marginally above average’. So the descriptions suggest the average is between 4 and 5, and contrary to the graph empathy isn’t normally distributed. He repeats the claim that empathy levels are normally distributed many times in the book.
• The same thing happens with his ‘systemizing’ spectrum: the graph on page 80 shows a normal distribution from 0 to 6 averaging 3, while the descriptions (page 80-81) suggest that the average is between levels 3 and 4, which isn’t what the graph says. It also means the distributions of empathy and systemizing are different, contrary to the impression given by the graphs looking exactly the same.
• Here’s a quote from page 48: “If you have empathy you will be capable of feeling guilt, while if you lack empathy, you won’t. This might make you think that guilt and empathy is one and the same thing: clearly this cannot be true, since a person can feel guilt (e.g. that they went through a red traffic light) without necessarily feeling empathy. So empathy can give rise to guilt but guilt is not proof of empathy.” (Italics added.) The two italicized phrases contradict each other, don’t they? He also uses the expression ‘clearly’ a lot when making claims which are not clear and may not even be true, which is well known to be an annoying thing to do.
• There is a diagram on page 31 illustrating the three forms of what he calls ‘zero-negative’ personalities. It looks just like a Venn diagram showing that zero-negative is the intersection of the borderline, psychopathic and narcissistic personality disorders. However, he wants to convey the quite different information that the three forms of zero-negative are those three personality disorders, and the diagram is pointlessly misleading.
• He defines empathy on page 12 like this: “Empathy is our ability to identify what someone else is thinking or feeling, and to respond to their thoughts and feelings with an appropriate emotion.” That’s fine: it’s a reasonable definition. But in chapter 6 (‘Reflections on human cruelty’) he says that it follows from the definition that people commit various acts harming people (his examples involve murders and terrorist bombings) must have their empathy switched off at the time of action. This isn’t true, because there is no inconsistency in feeling an emotion and acting in spite of it. The line of thought that empathy is incompatible with some actions, even some praiseworthy actions (see note iv), appears frequently, especially in chapter 6, and it isn’t right.
• Here is a quote from page 26: “…we react in a very sensory way when we identify with someone else’s distress. This clear brain response is telling us that even without any conscious decision to do so we must be putting ourselves into the other person’s shoes, not just to imagine how we would feel in their situation, but actually feeling it as if it had been our own sensation. No wonder we wince involuntarily when we see someone else get hurt. Of course, not everyone will have this strong empathic response to such emotionally charged situations. If our somatosensory cortex is damaged or temporarily disrupted, our ability to recognize other people’s emotions is significantly diminished. Surgeons may, for example, be well suited to their job precisely because they don’t have this emotional reaction, a prediction that was confirmed by Yawei Cheng who found that physicians who practise acupuncture show less somatosensory cortex activity while watching pictures of body parts being pricked by needles.” (Italics in original.) This suggests that a relaxed response to needles makes people more likely to go into acupuncture. He doesn’t explain why he doesn’t think the causation might be the other way round.
• His definition of truth (pages 77-78) appears to be formulated to annoy philosophers. “Philosophers and theologians have long debated what we mean by truth. My definition of truth is neither mystical, nor divine, nor is it obscured by unnecessary philosophical complexity. Truth is (purely and simply) repeatable, verifiable patterns. Sometimes we call such patterns ‘laws’ or ‘rules’, but essentially they are just patterns.” I am a philosopher, and I am annoyed.
• His view is that the variation in incidence of zero-negative personality types is explained in significant parts both by variation in genes and by variation in upbringing. This may well be true. He presents evidence for the genetic component (from twin studies and gene testing), but his evidence for the upbringing component seems to be that the parents of zero-negative people disproportionately often mistreated them. Such a correlation would (of course) result from a genetic connection even if there was no causal contribution from upbringing. Perhaps he has evidence that upbringing makes a difference, but it isn’t in the book as far as I can tell. This makes it irritating when on page 89 he says “we have seen bucket-loads of evidence for the importance of early experience”.
• While he does present actual evidence for the genetic contribution, he also says this (pages 88-9): “there are parents who used the empathic, non-authoritarian style of parenting, discussing things reasonably with their child, yet their child still turns out to be a psychopath. Equally, we all know individuals who have thrived despite growing up in difficult environments… [Professor Dante Cicchetti] is proof that growing up in what James Blair calls a ‘dangerous and criminogenic’ environment does not totally determine your outcome. In his studies he found that as many as 80% of children who suffered abuse or neglect went on that [sic] to develop ‘disorganized attachment’. But clearly it takes more than a harsh environment to make a psychopath. There must be a genetic element.” Well, must there? It seems to me that this argument is easily parodied: disproportionately many people without access to clean water die in infancy, although not all do, and not all people who die in infancy lack clean water. Would this mean there was a genetic component? There could be a genetic component, but the rest of the variation might also be down to non-genetic chance factors. Drinking the wrong bit of water, perhaps. Or in the original case, having an encounter with the wrong person on the wrong day. Witnessing the wrong murder. Having the wrong friends. The possibilities are endless. These chance factors – what I understand the pros refer to as the unique environment – could explain the rest of the variation, without a genetic contribution. If I know this, Professor Baron-Cohen should know this.
• Here’s his guess at why empathy is distributed as it is (page 128): “Presumably the reason that empathy is a bell curve (with the majority of people showing moderate rather than high levels of empathy) is because moderate empathy levels are most adaptive.” He goes on to tell a story about why this might be. But does he really think that when evolved traits are normally distributed we can presume that moderate levels are most adaptive? Does he think this applies to intelligence, beauty, physical fitness, height, fertility, resistance to disease and so on? Perhaps these things aren’t normally distributed, but I’m fairly sure some of them are. Normal distributions appear all over the place, and they don’t need explaining in terms of the mode being the most adaptive. Again, if I know this, so should he.

I could go on, but that’s enough. Perhaps some of my criticisms could be given satisfactory responses, but I'd be astonished if there wouldn't still be a lot for him to think about. Simon Baron-Cohen is a professor at Cambridge and has been in the business for thirty years. He shouldn’t be making mistakes like these. It’d be nice if you could just come across a book like this, write the author off as a hack, and ignore them in future. You can’t do that, though. People trust experts. Laypeople trust them, of course, but academics in related fields trust them too, because you can’t be an expert in everything. We trust mathematicians that the proof of Fermat’s Last Theorem doesn’t have any errors in it (anymore), and Steven Pinker (in chapter 18 of The Blank Slate) trusts Simon Baron-Cohen [see correction]. There’s a lot of division of labour in our collective search for truth, and misinformation spreads. Of course people make mistakes, but if experts took a reasonable amount of care in their work then we could trust that their simple mistakes were reasonably rare. But it seems they don’t, and it seems we can’t.

UPDATE 17/08/12: CORRECTION

I said Steven Pinker trusted SBC in the chapter of The Blank Slate on gender. Sorry about this: I misremembered. He doesn't mention him there at all. In that chapter and others he relies on the work of a lot of experts in fields he isn't an expert in, so that part of the point still stands. SBC isn't one of those experts though, and is only briefly cited in the book, for his work on autism. Pinker and SBC do both hold the view that the average man isn't psychologically interchangeable with the average woman and that this is probably partly biologically caused, based on some of the same evidence. SBC's views on this (see especially his The Essential Difference) are however much more involved than anything Pinker defends in The Blank Slate. I might talk about TED in another post.

Who can you trust?

Regular readers will know that not so long ago I read and very much enjoyed Steven Pinker’s The Blank Slate, which is about the influence of genes on human psychology. That isn’t my area of expertise, but he seemed to be on the level. His reasoning was mostly reasonable and his evidential claims were backed up by sources which sounded reputable enough. The first half of chapter 18 was about psychological differences between the sexes. He argued that there was strong evidence that some psychological traits are correlated with sex and some evidence that genetic differences between men and women contribute to these correlations. As I say, he seemed to be on the level and know his stuff, so I believed him.

Last week I got some conflicting signals, though. I read Cordelia Fine’s Delusions of Gender, and I enjoyed that a lot too. Fine’s book is about the claims made about psychological differences between men and women, both by scientists and popular writers drawing on the work of scientists. I suppose if I was summarising its claims in bullet points, I’d pick these: 

• There isn’t as much evidence for psychological differences between the sexes as a lot of people make out.
• A lot of the research into this sort of thing is done very badly.
• A lot of the popular writers either misinterpret or wildly extrapolate from what evidence there is, and sometimes just make things up.
• The hypotheses getting tested tend to be based on stereotypes.
• There’s no shortage of places to look for non-genetic explanations for the differences that have been found.

Fine seemed to be on the level just as much as Pinker did, and what she said was largely pretty persuasive. Since she went into far more detail about this specific issue than Pinker did, I suppose my credences are currently balanced in her favour, and my trust in the other 20½ chapters of Pinker’s book is correspondingly undermined. Mostly though, I just don’t know what to think. Fine goes into far more detail about the methods of the research she disagrees with than those of the research she uses to support her positive claims, so I’ve no way of knowing that I won’t read another book in a few months’ time which critiques that just as severely. If she had gone into as much detail about it all it would have doubled the length of her book though, so I can kind of see why she didn’t.

I like reading non-fiction, and I particularly like reading science books pitched at about the level Pinker’s and Fine’s books are pitched at. But I sometimes wonder why I bother. I’m trying to learn but if what I end up believing depends on which persuasive-sounding books are entertainingly written and easy to get hold of, then I’m not learning at all; I’m just making myself an unwitting vehicle for the memes I happen to get infected with. That’s no good. If all I’m going to learn from reading non-fiction is that scientists disagree with each other just as much as philosophers do and nobody really knows anything about anything, then maybe I’ll just read PG Wodehouse all the time.

Experts

It’s seldom comfortable siding with the ex-mistress of a knight of the Garter, but a couple of days ago I was listening to Nicky Campbell’s radio show and that’s what happened to me. A man called Delroy Grant had just been convicted of some serious crimes and Campbell had taken this as a cue to have a nature/nurture debate. I’m currently reading and very much enjoying Steven Pinker’s The Blank Slate, which is about the genetic contribution to human psychology, so naturally I was interested.

Edwina Currie was on the show and thought that some people were born bad, and a psychologist called Martin had called in saying she was wrong and that people’s psychology resulted mostly from childhood experiences. Here’s some of what they said:

Martin: ‘Our states of mind aren’t born in us; obviously our brains are very ready to take on eighteen years of experience but... our brains are learning machines.’

Nicky tries to say something

Edwina: ‘No, Nicky, no, Nicky, we readily accept these days that we have blue eyes or brown eyes because we’ve inherited them...’

Martin (talking over her): ‘But that is genetic... there’s no evidence...’

Edwina: ‘...our hair colour, our height... yeah but how can it be so obvious that our physical characteristics are one way and then say the genes that control our mood, our attitudes...’

Martin (talking over her): ‘Genes don’t control our psychological characteristics.’

Edwina: ‘...oh but they do! My goodness of course they do!’

Martin: ‘You’re not an expert in this area; you’re a politician!’

They carried on for a while, and UK readers can listen to it here if they’re quick. (Edwina comes in at about 43:50.) Now I’m not taking sides here in the debate about whether violence begets violence or whether people with a genetic tendency to be violent beget people with a genetic tendency to be violent. What I didn’t like was that Martin seemed to be using his position as an expert to dismiss the other side of a genuine controversy. (I don't want to misrepresent Martin, and he did go on to admit that genetics did affect psychology in some cases, but this is how it came over at the time.) He asked Edwina for some evidence, but he’ll already know about the evidence from studying twins and adopted children and what have you, and what’s more he’ll know why he doesn’t think it shows that genes control psychological characteristics and why some of his colleagues do.

Now if I was calling in to Nicky Campbell’s show to talk about philosophy I wouldn’t dream of representing it as the expert view that moral statements attempt to state facts, that conceivability doesn’t even nearly entail possibility or that there's nothing more to the meaning of a proper name than its syntax and its referent, even though I think all these things. If Edwina said that it was obvious that ‘Clark Kent’ and ‘Superman’ weren’t synonymous I wouldn’t dismiss her as a mere politician. I’d know full well that plenty of experts disagree with me and agree with her.

The point of listening to experts is that they know what the state of the debate is. They know what’s controversial among people who know and what isn’t. They know what's a sensible way of approaching a question relevant to their field. If someone exploits the fact there aren’t any other experts around to push a controversial line then they’re misrepresenting their expertise and abusing our trust. If that’s how they behave then people won’t trust them, and if we can’t trust experts then we’ll benefit a lot less from having them around. Nobody wants public debate to be conducted exclusively by ill-informed laypeople trading ill-informed opinions, but if experts aren’t careful not to misrepresent things then that’s what we’ll be left with.