Yes, you can generally trust issue polling — but there’s a catch

A new study of polls on hundreds of state ballot initiatives reveals no bias towards liberals or conservatives, but rather a consistent underestimation of the status quo

There is a status quo bias in American politics. Most people think of the present as the ideal, and when the time comes to vote for reforms, they choose to vote against policies or politicians that promise to move the politics too far to the left or to the right. That’s one reason Bernie Sanders can’t win a party nomination for president, and why Ted Cruz would probably lose the general election if he did.

A new study finds that ballot initiatives may also suffer from status quo bias. My friends Jonathan Robinson (a data scientist and general smart guy for Catalist, a voter file company), Christopher Warshaw, and John Sides (political science professors at George Washington University and Vanderbilt, respectively) have just released new research showing polls overestimate support for ballot initiatives that are too liberal or conservative. Because writing about whether we can trust the polls is central to my work, I want to share some graphs from their paper in this short blog post today.

First, a note on methods. The team took results of statewide ballot initiatives since 1958 and compared them to estimated levels of support from a MRP model that turns a bunch of national polls into estimates of opinion at the state level. The statistical magic going on behind the scenes is that the researchers know the urbanity, race, gender, and education of both the people who took the poll and the people who live in each state, so they can train models to predict attitudes among the former onto the latter. They also know whether states tend to be richer or poorer and less or more conservative than others.

The plot below shows their estimates of the share of people who favored particular state-level ballot initiatives, according to their models, and the results of the actual ballot initiative.

You can see there is a general tendency to overestimate support for popular policies and understate support for unpopular ones. Put another way, the public is both less liberal and less conservative than polls reveal; the bias is towards the middle. The graph above muddies this point a little. Because conservative positions are less popular, they tend to fall on the left side of the chart, making it look like polls systematically favor liberals. But the below chart makes it clear this is really more of a mean-reversion issue. When conservative issue positions are popular, they share a similar fate as liberal ones; and vice-versa:

The researchers have four primary takeaways:

1. There is a strong (though not perfect) correlation between public opinion in polls and ballot initiative results

2. There is no global bias (toward liberals or conservatives) in initiatives vis-a-vis polls

3. Suggestive evidence that error/bias in relationship between initiatives and polls driven by:

   1. Orientation of polling question

   2. Change in the direction of status quo in initiative

As someone who cares a lot about what the nation as a whole thinks, these results are reassuring. It fits the growing body of evidence that issue polls are not substantially biased toward or against Democrats or Republicans.

But the new research does introduce a few (severe) caveats when covering state-level policy preferences. Left-leaning ballot initiatives, such as for gun control and marijuana legalization, do consistently worse at the ballot box in more Democratic states. Conservative proposals, correspondingly, are less popular than they appear in very red states. Reporting on state-level ballot initiative polls will have to take these dynamics into account.

Robinson, Warshaw, and Sides, for their part, plan to do more research into the source of these large biases towards the status quo.

. . .

I should add that there is one important problem with using ballot initiatives to study the accuracy of issue polls. This could explain part of the observed “status quo bias” the authors observe in their MRP estimates. That bias is that the population of adults in a state, which polls sample, is different from the population of voters who actually turn out for the ballot initiative. From other research (some of which is my own), we know that all adults nationwide tend to be at least a little more liberal than the electorate.

However, the adults-voters disconnect would only explain why the liberal issues (eg gun control) get overestimated in the data, not why popular conservative issues would be too. The operating hypothesis has to be that voters are more ideological than voters, which doesn’t really jibe with a lot of other research looking at the relationship between participation and attitudes. So the puzzle persists.

To throw one more wrench in the pool of explanations: the observed relationship here (a status quo bias) is the opposite of what we’ve observed in both regular polls and MRP models for presidential candidate vote choice, where polls tend to underestimate the dominant party in a given state. Weird!

Overall, this leaves me with one important question: should policymakers listen to the polls, which might offer a true quantification of what people want in the abstract, eg in an ideal world, or should they put the question to the people to decide in a referendum to decide the future of our real world? Expressed slightly differently: which is the “better” expression of what the public really wants: polls or ballot initiatives?

Given the strong status quo bias in ballot initiatives, Robinson, Warshaw, and Sides may just stumbled upon a big reason why they don’t actually improve representation all that much.

. . .

PS, the authors included this very entertaining comic with their research. This is proof enough that even academics can have a sense of humor.