Election polling on the wrong side of maybe
The misguided reason why so many people are so quick to dismiss polling
Friends,
Last week I wrote you a newsletter arguing that we should lower our expectations for pre-election polls. That was not an argument that polls are bad per se, but that spot-on election predictions in 2008 and 2012 gave many people a false sense of how accurate surveys can be. In our modern era of medium-turnout, polarized, competitive elections, polls will scarcely give us confidence over who will win an election.
Pollsters face a related problem in media coverage of their estimates — or, in the preferred lingo, “snapshots.” After the 2008 and 2012 elections, journalists reported that forecasters such as Nate Silver “called” 49 and 50 of 50 states correctly, by which they mean that pollsters gave the eventual winner of the contest a lead in the polls, by however much.
But this is not how polls work! And it’s not really what forecasters are doing either. Continuing to cover them in this way plays up the role of some heuristics that damage our collective understanding of what surveys are doing in the first place.
Editor’s Note: This is a paid post for premium subscribers. If you are a subscriber and have friends or family that you think might learn something from this post, you should feel open to forward it to them regardless of their membership status — but please also encourage them to sign up for posts themselves by clicking the button below!
See, articles about polling (especially after the fact) often fall prey to what gamblers and forecasters call the “wrong-side-of-maybe” fallacy. Meteorology offers some clues as to how it works. Imagine that you are readying to leave your house for the day and check the weather on your smartphone. If your weather app says there is a 51% chance of rain, you are probably going to bring your umbrella, implicitly assuming that it is going to rain outside — or at least that rain is so likely that you should be prepared to get wet if you leave your umbrella beside your front door. But what about a 40% chance? Maybe you would forgo the extra item. That’s the fallacy, that when something appears “likely” to occur by landing on the wrong side of “maybe,” people will psychologically prepare for it as if it is very likely — but otherwise will not.
Probabilistically, there was likely a similar dynamic at play in 2016. Although the polls indicated that Hillary Clinton was on the “wrong side” of maybe, they did not guarantee her a victory. Nate Silver gave her just a 71% chance of winning in the end. And though it was a far cry from 0, people were no doubt caught off guard by Tump’s 29% chance of winning coming true.
And in 2020, though Joe Biden was much likelier than a “maybe” to win overall, the polls ended up being “wrong” in both Florida and North Carolina, giving him leads of 3 and 2 points that were not realized. But that’s not actually a good way to think about polls. What they were really doing was pointing to a close race, with vote shares for both candidate within the margins of error with each other — rather than simply showing Biden “up.”
This represents a weakness of communication for the polls. Although it is commonly known that electoral predictions can be wrong, pollsters’ numbers tend to be repeated with little or no emphasis on uncertainty. “Biden is up by 3 in Florida,” someone may have been heard saying in the week before November 3rd 2020, “he’s got this.” The more accurate statement — that Biden would very probably win, but polls had been wrong before and could be even more errant in the future — tends to garner fewer headlines and turn fewer heads.
If we want to be smarter about the polls, and better represent what their measurements are actually telling us, we should stop reporting on their estimates as binary predictions. A more radical proposition would be to stop using the language of margins — “Biden +3, Trump +2, etc” and instead mention vote shares and margins of error for each candidate. Either would go a long way to restoring trust in an industry that dramatically needs the boost, and deserves more credit than it gets.
As a survey researcher, my rule of thumb for a single poll was that expected error was triple the random error, so that 1.96 s.d. was about .6745 s.d., or the reported 95% confidence interval was actually about 50%. At 50%, it' was an even money bet whether results would be within the 95% random confidence interval. For an average of many polls over time, the random component basically disappears (see the Dornsife USC panel poll), leaving at most the expected two-thirds that isn't random. But we don't know much about the distribution of this non-random error. If it followed the normal distribution and paralleled the random error but just didn't average out, then the estimated 95% confidence interval would be about an estimated 80% interval for non-random error, more or less the type of interval you and 538 were estimating. But once we rightly junk the random error basis of calculations, as we do with averages, the few historic episodes of presidential elections with many unknown base distributions for variables provide no defensible grounds for any estimates of reliability. Yes, we want to estimate as a practical matter, but statistics, estimated error, and confidence intervals -- is there really much basis for saying that postwar economic measurements provide some reliable basis for estimating voting error likelihood in a covid-19 economy with stimulus?