Rule One: Search Your Feelings
Our emotional reaction to a statistical or scientific claim isn’t a side issue. Our emotions can, and often do, shape our beliefs more than any logic. We are capable of persuading ourselves to believe strange things, and to doubt solid evidence, in service of our political partisanship, our desire to keep drinking coffee, our unwillingness to face up to the reality of our HIV diagnosis, or any other cause that invokes an emotional response. But we shouldn’t despair. We can learn to control our emotions—that is part of the process of growing up. The first simple step is to notice those emotions.
When you see a statistical claim, pay attention to your own reaction. If you feel outrage, triumph, denial, pause for a moment. Then reflect. You don’t need to be an emotionless robot, but you could and should think as well as feel. The cognitive reflection questions invite us to leap to the wrong conclusion without thinking. When was the last time Donald Trump, or for that matter Greenpeace, tweeted something designed to make you pause in calm reflection?
Today’s persuaders don’t want you to stop and think. They want you to hurry up and feel. Don’t be rushed. When we encounter a statistical claim about the world and are thinking of sharing it on social media or typing a furious rebuttal, we should instead ask ourselves, “How does this make me feel?”
Rule Two: Ponder Your Personal Experience
If we don’t understand the statistics, we’re likely to be badly mistaken about the way the world is. It is all too easy to convince ourselves that whatever we’ve seen with our own eyes is the whole truth; it isn’t. Understanding causation is tough even with good statistics, but hopeless without them.
And yet, if we understand only the statistics, we understand little. We need to be curious about the world that we see, hear, touch, and smell, as well as the world we can examine through a spreadsheet.
Tim’s second piece of advice, then, is to try to take both perspectives—the worm’s-eye view as well as the bird’s-eye view. They will usually show you something different, and they will sometimes pose a puzzle: How could both views be true? That should be the beginning of an investigation. Sometimes the statistics will be misleading, sometimes it will be our own eyes that deceive us, and sometimes the apparent contradiction can be resolved once we get a handle on what is happening.
Rule Three: Avoid Premature Enumeration
When we are trying to understand a statistical claim—any statistical claim—we need to start by asking ourselves what the claim actually means.
Measuring infant mortality, at first glance, means doing something sad and simple: counting the babies who died. But think about it for a moment and you realize that the distinction between a baby and a fetus is anything but simple—it’s a deep ethical question that underlies one of the most acrimonious divides in US politics. The statisticians have to draw the line somewhere. If we want to understand what is going on, we need to understand where they drew it.
Statisticians are sometimes dismissed as bean counters. The sneering term is misleading as well as unfair. Most of the concepts that matter in policy are not like beans; they are not merely difficult to count, but difficult to define.
The solution, then: Ask what is being counted, what stories lie behind the statistics. It is natural to think that the skills required to evaluate numbers are numerical—understanding how to compute a percentage, or how to disentangle your millions from your billions from your zillions.
Before we figure out whether nurses have had a pay raise, first find out what is meant by “nurse.” Before lamenting the prevalence of self-harm in young people, stop to consider whether you know what “self-harm” is supposed to mean. Before concluding that inequality has soared, ask, “Inequality of what?” Demanding a short, sharp answer to the question “Has inequality risen?” is not only unfair, but strangely incurious. If we are curious instead, and ask the right questions, deeper insight is within easy reach.
Rule Four: Step Back and Enjoy the View
In the crazy early days when the COVID-19 coronavirus went global, Scientific American admonished journalists, “Facts about this epidemic that have lasted a few days are far more reliable than the latest ‘facts’ that have just come out, which may be erroneous or unrepresentative and thus misleading . . . a question that today can be answered [by] only informed belief may perhaps be answered with a fact tomorrow.”
Sound advice, and not just for journalists but for citizens, too. So however much news you choose to read, make sure you spend time looking for longer-term, slower-paced information. You will notice things—good and bad—that others ignore.
Rule Five: Get the Backstory
Darrell Huff’s How to Lie with Statistics describes how publication bias can be used as a weapon by an amoral corporation more interested in money than truth. With his trademark cynicism, he mentions that a toothpaste maker can truthfully advertise that the toothpaste is wonderfully effective simply by running experiments, putting all unwelcome results “well out of sight somewhere” and waiting until a positive result shows up.
That is certainly a risk—not only in advertising but also in the clinical trials that underpin potentially lucrative pharmaceutical treatments. But might accidental publication bias be an even bigger risk than weaponized publication bias?
Rule Six: Ask Who Is Missing
Unless we’re collecting data ourselves, there’s a limit to how much we can do to combat the problem of missing data. But we can and should remember to ask who or what might be missing from the data we’re being told about.
Some missing numbers are obvious—for example, it’s clearly hard to collect good data about crimes such as sex trafficking or the use of hard drugs. Other omissions show up only when we take a close look at the claim in question. Researchers may not be explicit that an experiment studied only men—such information is sometimes buried in a statistical appendix, and sometimes not reported at all. But often a quick investigation will reveal that the study has a blind spot. If an experiment studies only men, we can’t assume it would have pointed to the same conclusions if it had also included women. If a government statistic measures the income of a household, we must recognize that we’re learning little about the sharing of that income within a household.
We must always ask who and what is missing. And this is only one reason to approach big data with caution.
Rule Seven: Demand Transparency When the Computer Says No
Big data is revolutionizing the world around us, and it is easy to feel alienated by tales of computers handing down decisions made in ways we don’t understand. I think we’re right to be concerned. Modern data analytics can produce some miraculous results, but big data is often less trustworthy than small data. Small data can typically be scrutinized; big data tends to be locked away in the vaults of Silicon Valley.
We should ask tough questions on a case-by-case basis whenever we have reason for concern. Are the underlying data accessible? Has the performance of the algorithm been assessed rigorously—for example, by running a randomized trial to see if people make better decisions with or without algorithmic advice? Have independent experts been given a chance to evaluate the algorithm? What have they concluded? We should not simply trust that algorithms are doing a better job than humans, nor should we assume that if the algorithms are flawed, the humans would be flawless.
Rule Eight: Don’t Take Statistical Bedrock for Granted
When a country picks and defends a team of skilled, professional, and independent statisticians, the facts have a way of making themselves known. When a country’s national statistics fall short, an international community of statisticians will complain. When an independent statistician is attacked or threatened by politicians, that same community will rally to his or her defense. Statisticians are capable of greater courage than most of us appreciate. Their independence is not something to take for granted, or to casually undermine.
As citizens, we need to look for that statistical bedrock. If we want to understand the situation a country is in—whether to inform our own decisions, or to hold our government to account—then we will usually start with the statistics and the analysis produced by third-party institutions.
Rule Nine: Remember That Misinformation Can Be Beautiful, Too
A good chart isn’t an illustration but a visual argument,” declares Alberto Cairo near the beginning of his book How Charts Lie. As the title of the book implies, Cairo has some concerns. If a good chart is a visual argument, a bad chart may be a confusing mess—or it may also be a visual argument, but a deceptive and seductive one. Either way, by organizing and presenting the data, we are inviting people to draw certain conclusions. And just as a verbal argument can be logical or emotional, sharp or woolly, clear or baffling, honest or misleading, so too can the argument made by a chart.
The data density of the graph is no guarantee that the data themselves are reliable: a graph that presents a few data points in a lighthearted fashion may be unimpeachable, while an intricate graphic may be saturated with bad data. Even if the numbers are solid, a graphic detailed enough to demand a coffee may be persuasive without also being informative
Rule Ten: Keep an Open Mind
A man with a conviction is a hard man to change. Tell him you disagree and he turns away. Show him facts or figures and he questions your sources. Appeal to logic and he fails to see your point.
This book has argued that it is possible to gather and to analyze numbers in ways that help us understand the world. But it has also argued that very often we make mistakes not because the data aren’t available, but because we refuse to accept what they are telling us
The Golden Rule: Be Curious
The cure for boredom is curiosity,” goes an old saying. “There is no cure for curiosity.” Just so: once we start to peer beneath the surface of things, become aware of the gaps in our knowledge, and treat each question as the path to a better question, we find that curiosity is habit-forming.
Sometimes we need to think like Darrell Huff; there is a place in life for the mean-minded, hard-nosed skepticism that asks, Where’s the trick? Why is this lying bastard lying to me?
But while “I don’t believe it” is sometimes the right starting point when confronted with a surprising statistical claim, it is a lazy and depressing place to finish.