Stuck at home like I am? Let’s chat about polls! Ask anything you want or just share something you read recently. Visit our most recent thread on polls for ideas or to remind me about unanswered questions. I’ll respond to everyone who comments before I go offline.
Ok, discuss!
EDIT 10:34 PM EDT: I’m going offline now—thanks all for the engagement! I think it might be a good idea to increase the frequency of these chats because I really enjoy them, and apparently some of y’all do too!
A couple of different questions. Respond to whatever interests you the most!
1. What are some weighting variables that polls do not currently use that you think could make a big difference. We can ignore feasibility for now. Would weighting towards religion increase accuracy? What about income or even something like social media usage?
2. What are your thoughts on moving away from traditional weighting schemes and using models directly. Using a model would allow us to share information both temporally and spatially. Rather than just using one poll we aggregate multiple polls. Do you think this is feasible and would it increase accuracy from current methods. What do you see as some of the dangers of this?
3. Do you have any ideas about how we could re-think traditional margin of error. Traditionally margin of error only considers sampling error. It does not consider inaccuracy in weighting variables, people's opinions changing or unseen confounding variables. Do you have any thoughts about a better way to convey potential error in a poll?
Pollsters sometimes do weight toward region and income, though not a lot of them. I don't think social media usage is that explanatory a variable. I do think pollsters should consider weighting by past vote, at least for those who said they voted last time.
On 3: This is a good question that deserves more thought. For MRP projects I've conducted, I have considered passing on the uncertainty in the voter turnout models to the margin of error, which is one possible way to do it, but this is (as you point out) still a pretty open question.
Pretty impossible to precisely anticipate the impacts of the virus, IMO, and I don't think asking who benefits from lower turnout is really the proper question during this outbreak.
I didn’t mean to sound crass. Just wondering what the thought process might be. It seems like moving to no fault absentee voting or vote by mail should be the takeaway from this crisis.
Yeah, I think that's clearly the right move after this. Crises like the covid-19 outbreak tend to open our eyes to structural policy and institutional issues.
How much of Trump’s approval/disapproval has to do with the state of the economy? Like if the economy goes into recession for a few months how much would that affect his numbers based on the data that’s out there?
Historically, have there been polls on P/VP posssible combos & how they might affect GE victory (eg Electoral College) chances? Have they been at all accurate, if so? And what was their margin of error? Or is this not a thing?
As a swing state & swing county resident who's done quite a bit of door knocking over the decades, those finding are pretty much spot on with what I've seen. I'm in NY19, where the House Dem primary election in '18 in a way mirrored this year's Dem primary. The freshman Dem had the most money, was the most moderate and they did many, many 'debates.' Both the more right leaning & more left leaning candidate (field of many) lost narrowly in the primary. Our incumbent R lost narrowly in the GE, as despite the rather bitter primary contest, all the Dems hated the GOP incumbent.
If the biggest flaw in 2016 polls is weighing by education and almost all pollsters fixed it this year (except MEMErson), any other factors we should trust GE polls? Besides margin of error.
Many of them don't have the time to develop the additional weights to correct for 2016's mistakes, I believe, or else they don't see the need. It's a pretty foolish decision not to do so, IMO.
First, I saw you (or maybe another days journalist) tweet a while back that head-to-head presidential polls last cycle were very close to the ultimate outcome. But you recently tweeted that Obama’s and Hillary’s polls at this point in their cycles had them winning by much greater margins than they did. How do you fit those together?
Second, how seriously should we take recent polls measuring approval of Trump’s coronavirus response? They align very closely with his approval ratings, but the worst of the outbreak is obviously yet to come.
How seriously should we take national / state level general election polls at this point? It seemed like they were quite often a function of name recognition in terms of which Dem candidates did better than others in 2019. How much should those polls influence our priors at this stage in the race?
Great question Zachary. Polls fielded this far before the election aren't that predictive historically, but in recent cycles they have performed quite well (errors of ~4 points on average at this point). Read this old blog post from me and a co-author: https://www.thecrosstab.com/2017/01/03/history-polling-error-us-uk/
History of polling: Polling has changed since 1935 (Gallup origins) in questions asked and in populations asked. For example, black voters weren't polled prior to 1980. https://www.nbcnews.com/politics/2020-election/journey-power-history-black-voters-1976-2020-n1029581. Thinking about polling this way, who gets asked what questions when, I am struck by the disconnect between polls and people's concerns, and between the mainstream media narrative derived from polls and the public's actual concerns.
For example, we have plenty of polls on race relations, but I am not finding polls on disparate access to resources. We have polls on racial resentment, but I don't find polls on public resentment of capitalists moving jobs offshore, getting tax breaks, getting Congress to provide beneficial legislation at the expense of the general public. Instead, we see polls and resulting narratives that distort the facts and result in electing republican tools of capital. Since 1968, we have elected 5 of the nation's worst presidents. I haven't done a review of the polling questions asked from 1968 on but I believe I will.
Nothing yet to share here, but an article coming soon I hope. I need to get on updating that analysis for recent primaries, though the calendar is pretty tricky going forward given the cancellations caused by the coronavirus.
The Data for Progress polls are mostly text-to-web, where they text people links to complete surveys online. Their accuracy is being primarily driven, I suspect, by the fact that people are selected from the voter file, allowing them to pretty effectively identify the likely voter universe. But there's a bit of luck involved too!
I used to run a discord server, but it was a bit ~weird~ with my staff writing gig. Substack allows me to communicate with y'all in a more journalistic, writer-to-reader type of way.
A couple of different questions. Respond to whatever interests you the most!
1. What are some weighting variables that polls do not currently use that you think could make a big difference. We can ignore feasibility for now. Would weighting towards religion increase accuracy? What about income or even something like social media usage?
2. What are your thoughts on moving away from traditional weighting schemes and using models directly. Using a model would allow us to share information both temporally and spatially. Rather than just using one poll we aggregate multiple polls. Do you think this is feasible and would it increase accuracy from current methods. What do you see as some of the dangers of this?
3. Do you have any ideas about how we could re-think traditional margin of error. Traditionally margin of error only considers sampling error. It does not consider inaccuracy in weighting variables, people's opinions changing or unseen confounding variables. Do you have any thoughts about a better way to convey potential error in a poll?
Pollsters sometimes do weight toward region and income, though not a lot of them. I don't think social media usage is that explanatory a variable. I do think pollsters should consider weighting by past vote, at least for those who said they voted last time.
Oh, yeah, and the other questions!
On 2: I support this idea and think basically that we should be pooling raw data and using MRP: https://medium.economist.com/would-donald-trump-be-president-if-all-americans-actually-voted-95c4f960798
On 3: This is a good question that deserves more thought. For MRP projects I've conducted, I have considered passing on the uncertainty in the voter turnout models to the margin of error, which is one possible way to do it, but this is (as you point out) still a pretty open question.
This is more voting than polling, but how much lower Do you think turnout will be tomorrow compared to 2016! Who do you think it benefits more?
Pretty impossible to precisely anticipate the impacts of the virus, IMO, and I don't think asking who benefits from lower turnout is really the proper question during this outbreak.
I didn’t mean to sound crass. Just wondering what the thought process might be. It seems like moving to no fault absentee voting or vote by mail should be the takeaway from this crisis.
Yeah, I think that's clearly the right move after this. Crises like the covid-19 outbreak tend to open our eyes to structural policy and institutional issues.
How much of Trump’s approval/disapproval has to do with the state of the economy? Like if the economy goes into recession for a few months how much would that affect his numbers based on the data that’s out there?
I've written that the relationship between the two has decoupled rather quickly over the past decade. I don't think Trump will suffer as much as past presidents if the economy slips into a recession: https://www.economist.com/united-states/2019/05/23/american-voters-dont-care-about-the-economy
Is there a list of polls that weigh by education and out if those which are online vs live caller?
I want to know which polls are doing the right things
I've been working on a list for my aggregation purposes at work, but it's not comprehensive yet.
I am excited to see it.
Historically, have there been polls on P/VP posssible combos & how they might affect GE victory (eg Electoral College) chances? Have they been at all accurate, if so? And what was their margin of error? Or is this not a thing?
I don't have a large historical set of hypothetical matchups, so I can't reach a verdict for you. But did you read this piece? https://www.vox.com/policy-and-politics/2020/2/25/21152538/bernie-sanders-electability-president-moderates-data
As a swing state & swing county resident who's done quite a bit of door knocking over the decades, those finding are pretty much spot on with what I've seen. I'm in NY19, where the House Dem primary election in '18 in a way mirrored this year's Dem primary. The freshman Dem had the most money, was the most moderate and they did many, many 'debates.' Both the more right leaning & more left leaning candidate (field of many) lost narrowly in the primary. Our incumbent R lost narrowly in the GE, as despite the rather bitter primary contest, all the Dems hated the GOP incumbent.
Ah, the power of negative partisanship!
Nope so thanks! I will now...
General Election Polls are accurate this year, right? I'm talking about the Arizona one today.
Impossible to say until the election actually happens!
If the biggest flaw in 2016 polls is weighing by education and almost all pollsters fixed it this year (except MEMErson), any other factors we should trust GE polls? Besides margin of error.
It's not clear to me that most pollsters have actually fixed this problem. See: https://www.nytimes.com/2017/11/06/upshot/after-a-tough-2016-many-pollsters-havent-changed-anything.html
ooooh... Why not, do you think?
Many of them don't have the time to develop the additional weights to correct for 2016's mistakes, I believe, or else they don't see the need. It's a pretty foolish decision not to do so, IMO.
*should NOT
Two questions (sorry to be that guy).
First, I saw you (or maybe another days journalist) tweet a while back that head-to-head presidential polls last cycle were very close to the ultimate outcome. But you recently tweeted that Obama’s and Hillary’s polls at this point in their cycles had them winning by much greater margins than they did. How do you fit those together?
Second, how seriously should we take recent polls measuring approval of Trump’s coronavirus response? They align very closely with his approval ratings, but the worst of the outbreak is obviously yet to come.
Thank you!
Hey. Yeah, I tweeted both those things. Some thoughts:
My most recent tweet probably overstated the error a bit. Commented on that here: https://twitter.com/gelliottmorris/status/1239632719396691973?s=20
My usual comment is that polls this early do a pretty good job at this point. That doesn't mean they're perfect, though. Just look at this graph: https://www.thecrosstab.com/post/2017-01-03-history-polling-error-us-uk/Figure4.png
Data* journalist. My bad.
How seriously should we take national / state level general election polls at this point? It seemed like they were quite often a function of name recognition in terms of which Dem candidates did better than others in 2019. How much should those polls influence our priors at this stage in the race?
Great question Zachary. Polls fielded this far before the election aren't that predictive historically, but in recent cycles they have performed quite well (errors of ~4 points on average at this point). Read this old blog post from me and a co-author: https://www.thecrosstab.com/2017/01/03/history-polling-error-us-uk/
History of polling: Polling has changed since 1935 (Gallup origins) in questions asked and in populations asked. For example, black voters weren't polled prior to 1980. https://www.nbcnews.com/politics/2020-election/journey-power-history-black-voters-1976-2020-n1029581. Thinking about polling this way, who gets asked what questions when, I am struck by the disconnect between polls and people's concerns, and between the mainstream media narrative derived from polls and the public's actual concerns.
For example, we have plenty of polls on race relations, but I am not finding polls on disparate access to resources. We have polls on racial resentment, but I don't find polls on public resentment of capitalists moving jobs offshore, getting tax breaks, getting Congress to provide beneficial legislation at the expense of the general public. Instead, we see polls and resulting narratives that distort the facts and result in electing republican tools of capital. Since 1968, we have elected 5 of the nation's worst presidents. I haven't done a review of the polling questions asked from 1968 on but I believe I will.
last month you dropped some early results of polling reliability and concluded "newer methods of polling the public are proving rather successful" ... https://twitter.com/gelliottmorris/status/1232123018002010113
Do you plan to write more as to how/why text (SMS) methods are faring the best? or any articles you can point to? Thnx!
Nothing yet to share here, but an article coming soon I hope. I need to get on updating that analysis for recent primaries, though the calendar is pretty tricky going forward given the cancellations caused by the coronavirus.
Interested in your thoughts on why the data for progress primary polls have been so accurate so far? What’s the method they’re using there?
The Data for Progress polls are mostly text-to-web, where they text people links to complete surveys online. Their accuracy is being primarily driven, I suspect, by the fact that people are selected from the voter file, allowing them to pretty effectively identify the likely voter universe. But there's a bit of luck involved too!
You should do this on discord or WhatsAp. I think that would be fun.
I used to run a discord server, but it was a bit ~weird~ with my staff writing gig. Substack allows me to communicate with y'all in a more journalistic, writer-to-reader type of way.
Every political idea started out weird. Benjamin Franklin and James Madison were weird freaking people.