In Iowa, a victory for new advances in polling

Polls conducted online, and even by text, performed even better than traditional phone polling

The fiasco in Iowa continues. It is now over 24 hours after the start of the caucuses and we have yet to receive a final count of the results. No word yet from the Iowa Democratic Party about when we will get the remaining 38% of precinct-level results (you can look at the 62% we do have online at the Washington Post).

But the results shouldn’t change that much between now and the final count, so I want to take some time to assess the accuracy of the polls released in the run-up to Iowa. The results are pretty revealing about both the current state and future of political polling.

To measure accuracy of each firm, I matched the current returns — Sanders 25%; Buttigieg 21%; Warren 19%; Biden 15%; Klobuchar 13% — with the last month of polls aggregated by FiveThirtyEight and compared each of the top 5 candidates’ projected vote shares in the polls versus their actual vote shares. I then took an average of those errors for each pollster. (In data analysis we call this the “mean average error” or MAE, just for whatever that’s worth.)

Here are the six pollsters with the lowest errors for Iowa:

• Civiqs: 3.7%

• Change Research: 3.8%

• NYT/Siena College: 4%

• David Binder Research: 4%

• Data for Progress: 4.4%

• YouGov: 4.4%

This list of pollsters is particularly revelatory because only one of them uses the traditional technique of calling voters by phone. Polls from Siena College/the New York Times Upshot performed admirably using this technique. But the rest of the polls on this list either field all of or some of their surveys online or via text message.

The performance of these “alternative” polling methods would have been unheard of even last presidential election cycle, especially for polls fielded via text message. Data for Progress, a progressive think-tank, deserves an honorable mention here for conducting the entirety of their final pre-Iowa poll by via text.

To achieve this level of performance on such a new technology is admirable, but more importantly, it’s reassuring. The public polling industry has been experiencing several in recent years. First is cost. The percentage of people who will take a poll over the phone continues to decline, increasing the number of calls that pollsters have to make to get a workable sample. Then there’s representativeness. The type of people that answer landline phones—the cheapest mode for phone polling—are older and whiter than the population at large. You can fix this with weighting, but only if you know what to weight on beforehand (and actually do it — see: 2016). And then there’s trust and legitimacy, which will be a harder battle to win.

The performance of online (and SMS text) polls before Iowa are reassuring on all these fronts. They are cheaper. They can use complex sampling designs—such as YouGov’s synthetic sampling and Change Research’s live target adjustments—to approximate random samples of the population (and collect data cheaply to try different approaches rapidly). And, if they continue performing well, they may help convince people that the future of polling is alright.

One caveat is that this is just one election. We need a sample size larger than one contest to know if patterns like these are going to stick.

***
I’m going to continue to think more about these issues as the primary continues. In the meantime, be sure to pass along your thoughts. And follow along with the tweed threat below, where I’ll post a final update to these numbers when Iowa finalizes its results.

G. Elliott Morris @gelliottmorris

Results aren't final yet but here's how the pollsters are doing in Iowa - First picture is errors by mode for polls conducted in the final month of the campaign - Second is errors by pollster for their final poll - Last is error over time

---

Editor’s note:

Thanks for reading my thoughts on this subject. And thanks for subscribing! Your membership adds up and makes all this newslettering possible (reminder: I do all this work independently). Please consider sharing online or with a friend; the more readers, the merrier! 

As always, send me your tips about what you’d like to read about next, or your feedback otherwise. You can reach me via email at elliott@thecrosstab.com or @gelliottmorris on Twitter.

—Elliott