How not to “unskew” the polls
Corrections to inflexible partisan benchmarks usually fail
I wrote a couple a months ago about why I think the theoretical motivation behind “unskewing” the polls is completely fine, despite what you may hear from other analysts. People that attempt to de-bias the polls are simply asserting that most have too many Democrats or Republicans in them, pushing the industry to overestimate support for one candidate. Since we have plenty of evidence that polls with more Democrats in them in 2016 artificially inflated Hillary Clinton’s projected vote margins both nationally and (especially) in key states, it’s reasonable to try to “unskew.”
But the problem is not the idea of unskewing, it’s that it’s usually performed poorly. The practice of de-biasing the data consists of (1) figuring out how many Democrats or Republicans were sampled in a poll; (2) figuring out how many Democrats or Republicans there should have been for that geographic region; and (3) adjusting the poll numbers to what they would have been if they had the “correct” partisan composition.
But in the second step, people usually go badly wrong. That’s because there’s no 100% accurate measurement of the “true” partisan breakdown of, say, Alaska—or really even the country as a whole. That’s partially due to partisanship being a fluid characteristic; if you create your benchmark as the partisan breakdown in a few high-quality polls from November 2016 (or from the exit polls even), the underlying baseline could have changed since then. There are also question wording issues; if you take the partisan breakdown from an online survey that asks partisanship in a different way than the poll you’re “unskewing,” you’re going to have a problem.
This means that you probably need a flexible baseline for your partisanship. It’s my guess where you’d get a high-quality one. Maybe the vote file, you’d say? That’s a good starting point, but again you might run into some issues in the differences with how the file and to-be-adjusted poll measure partisanship. Or maybe you take a rolling average of the last few polls from high-quality, non-partisan outlets; say, the Pew Research Center, Gallup and the New York Times. That’s probably the most reasonable benchmark, actually—but it won’t buy you any mileage at the state-level.
That means that people who are unskewing are doing it with sub-par partisan benchmarks. That’s the procedure for one fella who popped up in my Twitter timeline this morning. “Perhaps,” you might contest, “unskewing is better than doing nothing at all. And if you can guess if polls are over-estimating one candidate, isn’t that better than doing nothing?”
As it turns out, no. The typical unskewing embodied by this man on twitter made polls worse than a replacement-level polling model. The tweet below contains some statistics showing that the absolute errors in predicted vote share were about 46% worse in the unskewed data than in a mixture of the raw polls and a reasonable benchmark in each state (that benchmark is equal to the states’ partisan leans in the last two elections, plus the predicted national vote this time).
But this unskew-er does have a point. Although his predictions were further “spread out” around the actual results of the 2016 elections (that’s what the second column above shows), their average raw error was closer to the result—and by a rather large amount. So, should we believe him?
I think not. I think we should glean from the second column of error above that this method is fraught with inconsistencies—even when it works, it makes the average state more off of the result of the election in absolute terms than just using the polls alone. When data scientists see such differences between the mean and root-mean-square error of a forecast, we usually conclude that it just got lucky one time and woudln’t the next.
And along those lines, it is worth noting that the same technique did not work in 2012 when polls underestimated Barack Obama. And I don’t think it’s a coincidence that the people doing these exercises are typically right-leaning. They tend to dismiss the polls that show Democrats ahead and embrace the ones that don’t. So unless you have evidence that Trump will beat expectations again in 2016, I think the unskewing will introduce more error in the models than not.
I am open, however, to better, more flexible methods of unskewing the polls. If you’re using reliable baseline that is allowed to change over time or with new data, that’s better than making assumptions about the “true” partisan composition of the electorate that just so happens to align with your ideological goals.