Now out! My book STRENGTH IN NUMBERS: How Polls Work and Why We Need Them

A one-of-a-kind book about the history of polling, mechanics of modern surveys, and role of polls in our democracy

Jul 12, 2022

Dear readers,

An event that once seemed stuck in the distant future is now upon us: the publication date for my new book STRENGTH IN NUMBERS: How Polls Work and Why We Need Them.

Click here to order your copy today!

In this post, I want to tell you a little bit about why I wrote it, and why I believe so strongly in the polls as a tool for democracy.

Why people need a book about polls

In late 2019, I decided to write a book to teach people everything I knew about public opinion polling and election forecasting. After all, there is intense consumer interest in forecasting elections. Surely that applies to understanding polls, too — and why they sometimes go wrong. As a journalist who covers public opinion and develops election forecasting models, I thought I was uniquely situated to explain political surveys to a broader audience (even though I am, admittedly, not the one doing most of the actual surveying).

That may sound crazy to you. Pollsters had some of their worst performances on record in the 2020 and 2016 elections. Errors were correlated across states in new ways that prompted some people to declare polling dead. Misfires were “catastrophic,” according to others. The view from New York and Washington is that the industry is teetering on the brink of irrelevance, or worse.

Now, these terminal diagnoses were obvious exaggerations. The fact is that polls in each year were a little above the historical average nationally, and in key states were a bit worse. But the public opinion survey has hardly degraded to the point where it cannot anticipate election outcomes or attitudes on issues of the day.

That is, however, not a justification to write a whole book. Reporters and pundits indeed expect far too much certainty from polls. That they are lazy when reporting on them and pass on distorted views of surveys to readers. And, yeah, they have been led astray partly by election forecasters, who marketed their ability to predict outcomes in every state with hyper-accuracy, but also by the pollsters, who have failed to explain the true uncertainty of their methods, and by their appetite for those predictions.

But to write a book about polling, I discovered I needed to start somewhere else. I did a lot of research. I dug into the archives of the academic public opinion and political science journals. I called all the pollsters I knew (and some I didn’t) and got their personal stories. I visited the libraries of presidents and politicians who were voracious consumers of polls. A new narrative emerged: Polling is not just about election prediction, but the democratic process itself.

I decided that a proper book on polls, not just election forecasts, would thus have to be divided into thirds. So the book first gives readers an understanding of the historical roots of public opinion and the philosophies of how our government is supposed to listen to it. Then, I explore various stories — from militia straw polls in the 1820s, to George Gallup's first "scientific" polls in the 1930s, to fabricated polling data in the Middle East and the rise of modern (eg online) polling — that show readers both the science and art of public opinion polls. And after all that, we look at how aggregation and election forecasting changed how people digest the polls, and we look forward: to how pollsters can fix the various problems pushing their science off course, and how people can still extract value from them regardless of those biases.

I hope that exposing readers to the normative importance of public opinion surveys may prompt them to give all polls, not just pre-election surveys, a second chance.

An artful science

So, go buy the book. But as you check out I want to say a little more about the 2016 and 2020 elections.

Polls suffer generally from something called “nonresponse bias,” their term for the chance that the people who are answering polls are systematically different than the population as a whole. This bias gets worse as response rates to a survey fall. In the 1970s, when telephone polls were first developed, response rates were at least 70% to any poll, and some firms got them even higher — to 80 or 90%, depending on how they selected samples, dialed numbers, and opened their calls. But response rates today are much lower — about 1% for most phone polls. And that means there is a lot of nonresponse.

To combat this issue, pollsters developed statistical algorithms that they can use to make sure the samples of people they talk to are demographically representative of the population on different traits. Using computer programs, the samples pollsters collect are adjusted so they have the right shares of respondents by race, age, gender, region, and the like. If they talk to too few low-income voters, for example, pollsters adjust their “weight” in the survey so that each interview counts for more than one “person.”

But there are a few problems with these adjustments. The choices of what to weight on are often made arbitrarily (even though more sophisticated methods exist). The precise percentage of each group that is present in the electorate is often estimated, with its uncertainty. The characteristics of actual election-day voters are elusive and ephemeral statistics, making it hard for pollsters to truly poll election-day “voters” instead of something else. And, finally, the weighting algorithms can often throw a poll off course, giving too much weight to just a few respondents — especially if the pollster tries to adjust for too many factors with too little data. And none of this even mentions the various ways human error can enter any scientific process.

Recent elections have shown that even if pollsters make all the correct decisions, they may still produce inaccurate estimates of the attitudinal breakdown of the electorate — especially on questions, such as vote choice, that have been polarized by demographics. The election in 2016 taught pollsters that weighting by a person’s education level is necessary to achieve a balanced sample, as white voters without college degrees weren’t as likely as other groups to answer surveys.

Then, in 2020, the Republicans who were most likely to support Donald Trump became even less likely to answer the phone. Post-election analyses taught us that even weighting by education is not enough. Pollsters also have to have the correct distributions of Democrats and Republicans in their samples as well. But those are unknowable quantities; no official survey tells us how many Trump or Biden supporters there are in the electorate. If pollsters wish to make these adjustments they risk addind additional error into the survey process.

This so-called “differential partisan nonresponse bias” has left the polling industry battered and bruised. Even though the average pre-election poll only overestimated Joe Biden's share of the vote by 2 points in 2020 — much too small an error to make a substantive difference in how leaders interpret issue polling, for example — we live in an era of close elections. That means that 2 points of error can make all the difference.

But political reporters don't see the polls this way. They see a polling industry that has previously produced hyper-accurate, laser-like predictions of election outcomes and now fails to do so. Because of their elevated expectations, they are primed to see polls today as "broken" or "useless." Yet this fails to take account of the reality of polling: that the 2008 and 2012 elections were unusual in the accuracy of their surveys, and the 2016 and 2020 elections were unusual in the industry's inaccuracy. That whiplashing of performance is a recipe for broken expectations.

Correcting these mistaken expectations for accuracy will be hard. That's especially true when election forecasters — like me! — present probabilistic forecasts for the horse race that often overshadow the uncertainty of our projections.

And that brings me to the book's real argument: a manifesto, so to speak, for the polls.

What polls are really good for

Above everything else, I argue people need to change how they view the polls. Current errors in thinking run along two axes.

First, people misunderstand the inherent uncertainty of the polls. The book talks about four types of error: Polls can be wrong because they talk to an unrepresentative set of Americans just by random deviation in who gets polled; because of persistent systematic differences in the characteristics of people who are available to be polled versus those of the population as a whole; because of the possibility that the people answering the survey are systematically different than the people not answering them; and because question-wording elicits biased responses from respondents.

To make matters worse, all of these extra sources of error are unaccounted for in the traditional margin of error that pollsters report — a fact few pollsters advertise, and fewer political reporters even acknowledge. I want people to see polls less as predictions and more as uncertain estimates. Part of that means doubling the margin of error. I explain why in chapter 1.

The book also has a chapter on election forecasts; how they came about, how they work both methodologically and journalistically, how they go wrong, and how we can make them better. I discuss how the increasing chances of uniform bias in the polls calls the entire aggregation business into question. There are serious statistical challenges in the underlying claim that removing the noise from a group of polls (which is what an aggregate does) can improve the signal those polls produce if they are all subject to similar degrees of bias. So here, I cover what the future of polling means for forecasters, too.

But second, and more importantly, I want the press to write more about the normative utility of the polls. My study of the history of polls and democracy tells us that public opinion is extremely important to elected representatives, government officials, party activists, and the public itself. Doing what the people want is, at the end of the day, the ultimate task of representatives in a democratic system. And to restate a point from earlier: because legislators and elected officials are much less sensitive to slight deviations in the polls than, say, election forecasting models are, small errors make much less of a difference in how they get used. A politician does not care if a policy has the support of 42 or 44 percent of Americans.

So, although I set out to write an academic record of what I know about the polls — primarily marketed to people who wanted to know more about public opinion or how election forecasts work — I ended up writing a general-interest narrative history of the polls, and a case for using them differently.

As polls are, after all, just another way of talking to the people, I think that is both appropriate and novel. If we can come to view polls as rough guides for improving representation in government, instead of hyper-accurate tools that produce laser-like predictions of voting behavior, imagine how much good we can do for the people.

A mirror for the body politic

When George Gallup and Saul Forbes Rae (Gallup’s research assistant) wrote their most famous 1940 book about the polls, they promised the American people they would be able to measure “The Pulse of Democracy.” They wrote a book previewing a groundbreaking tool of government that leaders could consult in a rolling referendum on the goings-on of the state, and advise them on the issues people cared about.

I think this imagery is powerful in many ways, but misleading in some others. History has shown that polls are not as precise as evoked by the image of a pulse-oximeter (that device you place on the tip of your finger to track your pulse). Taking one's heartbeat is a straightforward task that requires no laws of random sampling or statistical wizardry to arrive at an estimate. Public opinion is not so easily measured.

But the people do have individual voices worth listening to. And when they are combined, those voices create the collective “public opinion” of our society. So I propose we talk about polls not as a pulse-oximeter, but as a mirror: one that we, the people, hold up to our collective self to see the shape of the body politic.

To continue the metaphor: That mirror does appear to be cracked and blemished in several places. Partisan nonresponse serves as the biggest injury to the mirror. Low trust in the pollsters also damages it.

Yet the mirror offered to us by public opinion polls is not completely shattered, and we ought not to treat it as such. Despite their errors as election forecasts, polls still give us a useful and reasonably accurate portrait of attitudes and American life. As the government drifts away from the desires of the popular majority, we should use that accuracy to our advantage.

Besides, even if the mirror is cracked, it is the best one — really, the only one — that we have.

I hope you'll all pre-order a copy (or two!) of the book. The story of the pollsters is entertaining — dare I say enthralling — and I think you will both enjoy and learn from them.

The link to WW Norton’s listing for the book is here. I would love it if you ordered it, read it, and sent me your thoughts.

Oh, happy day!

Elliott