The Value — and the Limitations — of Modern Political Polls

Pundits, partisans, and others have written much during this election season — and, increasingly, between election seasons — about polls and polling. Most of what has been written lately has been critical of polls and polling. Some have go so far to suggest that the polling industry is doomed.

This criticism is not new. Anyone who has seen the famous 1948 photo of Harry Truman holding up the newspaper proclaiming that “Dewey Defeats Truman” knows that journalism and polling have a troubled relationship. Some things have not changed since 1948. Journalists and pollsters still sometimes “get it wrong.” But the reality is that polling has come a long way since 1948, and with all due respect to those predicting the end of the polling industry, I’d like to suggest that this multi-billion dollar industry is here to stay. Why? Because those who actually pay for the polls, and use the results of the polls, know their enormous value.

Indeed, one of the key ideas of this essay is that it is not the pollsters who got it wrong, but the journalists who reported on the polls — the same ones who are now saying that pollsters got it wrong as a way of deflecting responsibility for their own malpractice.

We’ll come back to that idea later, but for now let’s just say that whether or not you believe the pollsters got it right or wrong in 2016, or 2020, it seems clear that political polling is here to stay. That’s why it’s important for us as citizens to know how to read them, what they can tell us, and — perhaps most importantly — what they cannot tell us.

So here’s a layman’s guide — you might call it a citizen’s guide — to understanding polls.

Polls Do Not Predict

First, it’s important to know that polls are not predictive. Assuming or asserting they are is perhaps the biggest way both journalists and the rest of us mis-read polls.

This misreading of polls as predictions is the key reason we tend to discount polls when the results of a particular poll doesn’t match the final result. This mis-reading often sounds like this: “The polls said Candidate A would win. Candidate B won. Therefore the polls got it wrong.”

So if polls are not predictions, what are they? The answer is that polls are snapshots of a particular population at a particular time. They do not tell you what that population thought yesterday or tomorrow. They tell you what that population thought on the day (or days) the poll took place. To use that result to predict the future is, for journalists and others, a nearly irresistible temptation. But we should resist it. Predicting the future is not what polls do.

Polls Over-count (or Under-count) Certain People

Another common objection to polls is that they “miss” certain people. They miss people who don’t trust pollsters and won’t answer their questions. Or they miss “shy voters” who don’t want to tell others what their preferences are.

To say that these phenomena never occur would be foolish. Of course they could occur. But do they? To answer that question, it’s important to understand how a pollster goes about his job.

First, pollsters develop models of a population they want to survey. Journalists often represent these models (imprecisely and simplistically) as “likely voters” or “all registered voters.”

But modern models include much more than that. The best models accurately represent the total population with the smallest possible sample. To use a simple example, if a population is 55 percent female and 45 percent male, you want your sample to have these percentages. That’s easy to understand, but it gets complicated when you are accounting for a dozen or more demographic characteristics that are sometimes at cross purposes.

What, for example, if a woman in your sample has a graduate degree and an income of $150,000? Is she representative of all women? Probably not. But she is certainly representative of some women. So you want her in your sample, but only in the same percentage as she is in the population as a whole. A pollster’s model says how many that should be. And when you’ve reached that percentage, the responsible pollster stops counting women like her. (To see an example of the parameters that Pew Research uses, click here.)

That’s why a poll of “1000 likely voters” doesn’t mean the pollster took a voter list and recorded the answers of the first 1000 people who answered the phone. This belief is common and often gets expressed this way: “I’ve never had a pollster call me” or “My candidate’s supporters won’t or don’t talk to pollsters.” Those statements may be true, but they are more-or-less irrelevant. Pollsters call until they get the appropriate number of samples from every parameter in their model.

And, these days, pollsters have to use a combination of land-line, cell-phone, and face-to-face interviews to get a sample that is truly representative of the population being surveyed. (For a more complete discussion of the relative advantages of various types of data collection, click here.)

Understanding The Margin of Error

It is, of course, possible that a pollster’s model is flawed. We live in a fallen world, and things break. But if you understand the paragraphs above, and you reflect for a moment on the massive amounts of data we now have about just about every aspect of our lives, you realize this kind of error is unlikely. That objection fundamentally misunderstands how a professional poll is done.

But even when these sampling errors do occur, they for the most part are accounted for within the margin of error, so let’s turn our attention to this important concept.

I said above that assuming polls are predictive is the most common error people make when reading polls. the second most common error is not understanding how the margin or error carefully limits what the poll is saying. Understanding the margin of error will prevent you from making false claims about a poll.

In simplest terms, the margin of error is a mathematical calculation that takes into account the possibility of the sampling errors we mentioned above.

Here’s a quick scenario to help us understand how sampling and how margins of error work:

If I flip a coin 10 times, it is highly improbable, almost impossible, that I will flip 10 heads or 10 tails. However, if I flip a coin 1-million times, it is highly likely that 10 consecutive heads or 10 consecutive tails will show up at least once in that set of 1-million flips. If I was sampling those 1-million flips, and my sample size was only those 10 heads (or tails), I would have a dramatically flawed understanding of the whole population. In short, I could not draw conclusions about the entire population of 1-million based on a sample size of 10.

But since such an anomaly happens very rarely, a sample size of 100 — even if it included that 10 all-heads or 10-all tales sequence — would look much more like the whole. A sample size of 1000 would almost eliminate the anomaly, because I would likely have some offsetting all-tails sequences in the sample.

So, the larger the sample, the smaller the margin of error, and the size of that margin can be reduced to a highly reliable mathematical formula. I will spare you the arithmetic. (Though if you want to dig into it, click here.) Instead, I’ll just jump to the answer: Most surveys today have about 1000 responses, because that creates about a 3 percent margin of error. A sample size of 2400 reduces the margin of error to 2 percent. Very few surveys go beyond 2400 because it requires a much larger sample size to reduce the margin of error only a very small amount. It’s just not worth the time, money, and trouble.

That’s why one objection to modern polling — “How can you survey just 1000 (or 2400) people and come up with reliable data?” — is a question that (to put it bluntly) reveals a great deal about the ignorance of the questioner, and says little that’s helpful about the poll itself.

So why is it important to understand the margin of error? Because understanding the margin of error will keep you from mis-reading a poll. It will keep you from saying what the poll is not saying.

The answer can be seen in another quick hypothetical but common scenario: Imagine that the day before election day, Candidate A has 51 percent and Candidate B gets 49 percent. The margin of error is plus or minus 3 percent. But when the election takes place, Candidate B actually wins the election. Everyone says, “See! The poll got it wrong.”

But remember two things we have said above. First, polls are not predictive, even polls done the day before an election. People are undecided, or they are still highly influenceable or ambivalent about their choice. So the poll did not, in fact, predict Candidate A would win. Remembering that polls are not predictive is even more important if Candidate A has 45 percent and Candidate B has 43 percent. That means 12 percent are undecided. These undecideds could swamp A’s modest lead.

And both of these phenomena are precisely what happened in the 2016 presidential race. The final polls predicted that Hillary Clinton would win the popular vote by about 3 percent. She ended up winning by 2 percent. But she won that popular vote by getting landslides in a few big states. In key swing states, she underperformed the polls, but by just 1 or 2 points — well within the margin of error. But that made the difference in the election. Again, it was not the polls that were wrong. It was an improper reading of the polls that created the error.

How To Read A Poll

But leaving Clinton and Trump behind, let’s go back to our hypothetical candidates and ask: what did the poll say? It said Candidate A would get between 48 and 54 percent of the vote and B would get between 46 and 52 percent of the vote. And that is exactly what happened.

But we don’t like that, so we tell ourselves (and journalists tell us, because reporting it accurately and completely is hard) just the two top-line numbers.

But in reality, I think you can now see, it was not the poll that got it wrong. The person who got it wrong was the person who drew a conclusion based only on Candidate A’s bigger top-line number.

It was that person, often a journalist, who did the predicting. Again, polls don’t predict.

I repeat that important point (that polls do not predict) because expecting a poll to predict an outcome is (again) the most common mistake people make in reading polls. To end where we began: Polls are snapshots of the conditions at a moment in time. Polls don’t predict. People do.

In closing, here’s an analogy that may help: If a ship is in the harbor, and I say, “There’s a ship in the harbor.” I am making a true statement. If you come tomorrow and that ship is gone, that does not make me a liar. Neither does my original statement, “There’s a ship in the harbor,” say anything about where the ship came from or where it is going. To judge my statement based on what it does NOT say is logically invalid and unfair.

So here’s the bottom line: Polls are (usually) not “wrong.” In fact, even the 2016 polls, which were widely maligned by the “hot takes” of pundits and partisans immediately after the election, have been analyzed and found to be more or less accurate.

But the 2016 and all polls are just very limited in what they tell us. So do not disrespect or disregard political polls. But — with apologies to my Latin-speaking friends — caveat lector: “Let the reader beware.”