The 1936 Literary Digest poll was off by 38 percentage points

George Gallup was off by 12. Think about that next time you see an election poll miss an outcome.

This is a quick post for posterity’s sake.

I have been traveling every couple of weeks since July to promote my book in some sort of event: academic talks, public speeches, panel appearances — you get the picture. It’s in the nature of book promotion that I give roughly the same presentation each time. I pull some numbers and tell some stories about how polls work, how they’ve changed over time, how accurate they are, what the industry’s future looks like, and what this all means not just for election prediction, but democracy too.

I gave a similar talk at Princeton’s Center for the Study of Democratic Politics yesterday, though this one had the benefit of hindsight on how the polls did in the midterms. That was an exciting development in the usual story. Polls, by the way, did pretty well — perhaps their best ever in Senate races.

That is a dramatic shift from the setting of the book’s introductory chapter in 2016 and 2020. This year's polls in competitive Senate races had an average error of about 2 points and no bias on average; in 2020, the average state presidential poll had an error of 4.5 points and overestimated Joe Biden’s vote margin by 4.2. The error is moving in the right direction! Hooray!

The point I make in the book is that polls have had some big misfires but have generally gotten better over time. Typically in presentations, I reference this graph, which shows the average absolute error and average bias of national polls going back to 1936. And, wow — polls have gotten a lot better over time. As of 2020, the average error in a national poll was about 2.5 percentage points for the last 20 years. It was higher for every 20-year period before that, going back to 1936.

But look at those early errors!

In 1936 the final Gallup poll had Franklin Roosevelt winning 56% of the two-party vote for president to 44% for his challenger, Alf Landon. The actual result of the election was 62-38 for Roosevelt — meaning he was off by 12 points on margin. Twelve! Over three times worse than the national polls in 2020!

Of course, as the story goes, George Gallup did significantly better than the unscientific “straw” poll run by the Literary Digest magazine. They had Roosevelt losing v Landon 43 to 57% — a 14 point margin. That means the Digest was off by 38 points. (This pretty much single-handedly killed the magazine.)

As it turns out, 38 is a lot larger than 12. It’s also a lot larger than 4 and infinitely larger than zero. I like to use this as a reminder to my audience that polling has come a very long way — and also to scare them into acknowledging that however wrong the surveys are in contemporary elections, from the historical perspective they could always be worse…