“Pollster who got it right in 2016 says...”

How to spot a bad poll when the narrative says otherwise

A reader named Neil sent along the following article last yesterday and asked “what do you think of this analysis?”

Here’s the article: “Pollster Who Got It Right in 2016: Michigan a Dead Heat”

The article is referencing a recent poll from the pollster Robert Cahaly, who does work for the right-leaning firm Trafalgar Group. Their poll showed Biden with a one-point lead in Michigan. That is a notable departure from the generally very blue numbers we’ve seen from other pollsters such as EPIC-MRA and Fox News over the past few weeks.

The piece argues that since Cahaly’s polls were better than others in 2016 (that’s kind of true; he predicted that Trump would win in Pennsylvania) then he must have some secret sauce that is making his polls better than everyone else’s.

To be sure, the Trafalgar Group’s polls are statistically no better than a replacement-level pollster. According to FiveThirtyEight’s pollster ratings (the overall grades from which I disagree with, but which present some good data) they have an average error of 5.6 percentage points, have called just 75% of the races they’ve polled correctly and overshoot Republicans by about 1 percentage point on average. They also make their phone calls via automated dialer, which introduces some massive coverage error and is generally a worse way to conduct your polls than via other methods.

Still, the article is focusing on the secret sauce; Cahaly did well in Pennsylvania in 2016 so there must be some trick up his sleeve, right? The author writes:

Cahaly’s survey, using the same methodology he employed four years ago but with an enhanced system for targeting likely voters, shows the race in Michigan as extremely competitive. The pollster also continues to see signs of “shy” or “reluctant” Trump voters in the electorate. Known as “social desirability bias,” it refers to the effect of respondents not telling the truth about whom they will vote for because they think their choice will be viewed unfavorably by others, including those conducting the survey. In a phone interview today, Cahaly said the social desirability bias he is seeing is “worse than it was four years ago.”

I think there are a few things wrong with this. First, there’s no real comprehensive proof that “shy Trump voters” actually exist. More than anything they are an appealing but empty answer to why polls went awry in 2016. If we were seeing social desirability bias among Trump voters we would see large differences in their willingness to express support for the president via phone polls (where people are sometimes more careful to admit their feelings to another human being) versus those conducted online (where they aren’t). Instead, both types of poll show the same (bad) numbers for Trump today.

Second, it’s unclear to me that the “neighbor question” (Cahaly’s method is to ask voters who they think their neighbors might vote for) is the way to combat this bias. Primarily I’m not sure how good people are at guessing their neighbor’s political leaning. That’s probably more of a product of where you live than how well you know your neighbor (lest they put up a yard sign for their choice candidate). And even if people can gauge that well there is still no comprehensive proof that I know of that this question is a proper tool to use.

Finally, I’ve seen nothing about Cahaly’s weighting procedure. He doesn’t publish it anywhere online. Automated (or IVR) polls are famous for getting samples that skew old and white. Looking at his most recent Michigan numbers, that definitely seems to be the case; they show that 67% of voters are over 50, which almost certainly is not true.

So, two points in conclusion: (1) It seems more likely to me that the method of the poll is responsible for its pro-Republican bias, rather than the secret sauce questions; and (2) in years when other polls have overestimated Democrats, that probably explains why Trafalgar’s polls look better—but come a time when the industry underestimates the party, they will look off.

…

Still, it may be reasonable for modelers to take what data they can get. As long as they acknowledge that all polls are not created equal — by, for example, accounting for whether firms typically lean to the left or the right — including biased data (like I suspect Trafalgar’s is) may still be somewhat useful when aggregated. Especially in a year when other, Democratic-leaning firms appear to be publishing wildly pro-Biden data, a check on their lean will be good in the end.