Hello from the Washington DC metro area, where it is a sunny and unseasonably warm, 75-degree day. I am writing to you from my back patio where our new puppy, a one-year-old lady Beagle named Maple, is splayed out in the sunshine. It is three days before the midterms. Oh, and happy Saturday.
I will write to the whole newsletter audience tomorrow with my near-final read on the midterms, according to the polls. Where do things stand in key states? Will the polls be wrong? How likely are Republicans to beat expectations? Is there a surge in turnout among women and young Democrats that the polls are missing? All that and more
Today, however, we return to an old format for the blog, the listicle, for some good links about polling that were posted over the last week. The takeaway from each of them is that polling is very hard — and that makes it inexact, too. We should not expect election predictions based on polls (or anything else for that matter) to be 100% accurate come Tuesday. Here’s why.
From Nate Cohn at the Times: “A Worrisome Pattern Re-emerges in Seeking Response From Republicans”. According to Cohn, white registered Democrats have been 28 percent more likely to answer calls for interviews than white Republicans, “a disparity exceeding that from our pre-election polling in 2020” (I recall Cohn has said elsewhere that the difference then was 25%, but don’t quote me on that). This comes after he wrote last month that only 0.4% of calls yielded a complete interview. The upshot here? Few people answer polls, and it’s hard to get those people to be politically representative of the country or key jurisdictions. Ine one district, Cohn says the response rate among Democrats was 70% higher than for Republicans—driven, most likely, by Biden voters (of all parties, but especially Democratic) being eager to talk to pollsters. My take? I am certainly more worried about polls overestimating Democrats than underestimating them.
From David Byler at The Washington Post: “The hidden, unsolvable problem with polls — and people who love them”. This piece is all about the crazy high expectations for accuracy that the public has for pollsters. It’s not just that consumers of polls want surveys to predict election results within a point or less of error, but that they want such. high degree of precision in an era of really competitive elections. Byler makes the point that even a 3-4 percentage point error in forecasting the national popular vote was not such a big deal when presidents typically won by landslide margins (6 or more points). But now, a 2, 3, or 4 point error is the whole ball game. My take? Another underrated factor that makes polling harder is the electoral college. It is not enough to have an accurate national survey. Pollsters have to create high-quality estimates in states, too— and they can be much harder to poll.
From Natalie Jackson in National Journal: “Polling averages can be more noisy, not less”. In this post, which is an admirably quick response to some recent bad punditry and even worse tweets about Republican pollsters “flooding the zone” with bad polls to bias averages in their favor, Jackson writes about how polling aggregation in the style done by people like Nate Silver is based on a flawed idea: that you can extract more signal from pollsters if you increase the weight on data from firms that have been more accurate historically. The article is paywalled but she screenshotted the main points for people to read freely on Twitter. My take? FiveThirtyEight currently does this weird thing where they increase the weight in their averages on polls with lower empirical error in their historical training set.
This last article is worth dwelling on a bit. FiveThirtyEight’s methodology is problematic for the reasons Jackson points out (the main one of which is that accuracy one year is only very, very weakly predictive of accuracy in the next year) but also because what an aggregate really needs is not polls with low error but polls with low bias. 538 currently rewards polls from the Trafalgar Group with an A- rating, for example, despite the fact that it has the 11th highest empirical bias toward Republicans out of nearly 500 pollsters they grade. Apparently they do this because the absolute values of Trafalgar Group’s biases have been lower than the absolute value of other pollsters’ biases, even though most of them do not send out polls that are reliably leaning toward either party.
Nate Silver provides no justification for this, and if I had to guess, it probably makes the aggregates less accurate, not more accurate. In my work for The Economist, we have found adjusting for past bias to improve aggregation a lot more than up-weighting pollsters with low historical error.
There’s one more thing to say. That is that polls are really, really noisy—so noisy that all this grading and adjusting and weighting is a very, very fraught exercise. You would need about 100 polls from each of two pollsters, for example, to know for sure if one firm has a lower empirical bias than the other. For most pollsters, we simply do not have a sample size that large. And that uncertainty happens before you try to forecast the values into the future!
To me, Nate Silver’s thinking that he can improve aggregation by increasing the weight on more accurate polls comes both from (1) a misunderstanding of survey methods (again, aggregates benefit more from low-bias polls than low-error polls, and Nate clearly has not considered this) and (2) a “big data” mindset I often encounter from people who do sports statistics, where they have a nearly unlimited domain of data to solve their problems and thus resolve to complicated weighting schemes to (try to) extract decimal points of additional accuracy out of their data. But with polls, this level of model massaging is simply vastly overcome by the inherent noise of the data-generating process.
That’s why I don’t take any of this pollster rating stuff all that seriously — and also why you should expect a hefty dose of error come Tuesday evening.
Leave your thoughts in the comments below. I would love to hear from you all.
Cohn has been writing about nonresponse bias and the tradeoffs in weighting by recalled past presidential vote for the last few days, which also strengthens my belief like yours that it's more likely we will see Democratic polling bias than the reverse this time again.
I'm worried that the nonpartisan industry standard pollsters will take a leaf from Trafalgar et al's book for future cycles if they get this year wrong again (ie, if the tossup Senate races end up being comfortable GOP wins). When the next inevitable Dem Blue Wave occurs (say 2026 or 2030 if we have unified GOP government by then), the polls may miss the outcome. Only this time, the GOP election denier faction may point to polling miss as evidence of election fraud as a pretext for shenanigans. Are you at all worried about the future of the industry becoming Trafalgar clones if the nonpartisan polls are wrong for 3 out of the 4 past cycles?
I see no paradox here, given the anti-GOP bias of most pollsters: "538 currently rewards polls from the Trafalgar Group with an A- rating, for example, despite the fact that it has the 11th highest empirical bias toward Republicans out of nearly 500 pollsters they grade."
With all Elliott's talk about Rep mo and bias in polls for Dems...Here is the last poll from the highest rated polling firm ABC/Washington Post....Talk to me Mr. Elliott
I think it is likely that we will see polls overestimate Democrats again. Republicans are releasing more internal polling that shows them ahead. I assume that Democrats' internal polls have similar numbers. The “flooding the zone” takes don't make sense to me. More likely, Democrats are getting their hopes up to high.
Cohn has been writing about nonresponse bias and the tradeoffs in weighting by recalled past presidential vote for the last few days, which also strengthens my belief like yours that it's more likely we will see Democratic polling bias than the reverse this time again.
I'm worried that the nonpartisan industry standard pollsters will take a leaf from Trafalgar et al's book for future cycles if they get this year wrong again (ie, if the tossup Senate races end up being comfortable GOP wins). When the next inevitable Dem Blue Wave occurs (say 2026 or 2030 if we have unified GOP government by then), the polls may miss the outcome. Only this time, the GOP election denier faction may point to polling miss as evidence of election fraud as a pretext for shenanigans. Are you at all worried about the future of the industry becoming Trafalgar clones if the nonpartisan polls are wrong for 3 out of the 4 past cycles?
I see no paradox here, given the anti-GOP bias of most pollsters: "538 currently rewards polls from the Trafalgar Group with an A- rating, for example, despite the fact that it has the 11th highest empirical bias toward Republicans out of nearly 500 pollsters they grade."
With all Elliott's talk about Rep mo and bias in polls for Dems...Here is the last poll from the highest rated polling firm ABC/Washington Post....Talk to me Mr. Elliott
https://docs-cdn-prod.news-engineering.aws.wapo.pub/publish_document/e1e76ca3-d201-4fe0-962c-2612ea4be47e/published/e1e76ca3-d201-4fe0-962c-2612ea4be47e.pdf
Hi Elliott,
I think it is likely that we will see polls overestimate Democrats again. Republicans are releasing more internal polling that shows them ahead. I assume that Democrats' internal polls have similar numbers. The “flooding the zone” takes don't make sense to me. More likely, Democrats are getting their hopes up to high.
I hope all is well!
Elliot
You're probably right....It's all about turn-out
I voted. I hope I get to do that again.