More evidence that polls are pretty good, actually — maybe even better than you've been told
Polls have tracked very close to the CDC's official record of how many people are vaccinated for covid-19, Pew says
The Pew Research Center published another useful defense of the polls yesterday afternoon, this time showing that self-reported covid-19 vaccination rates in surveys from 19 different polling organizations have strayed very little from the official rate from the Center for Disease Control since December last year. “Polling estimates of the adult vaccination rate have been within about 2.8 percentage points, on average, of the rate calculated by the CDC”, Pew reports. “Around a quarter (22%) of the polls differed from the CDC estimate by less than 1 percentage point.”
They include this excellent graph:
And while the average self-reported vaccination rate is closer to the CDC benchmark than polls were to the results of the election — aka lower bias in the aggregate, especially compared to state polls — the average absolute deviation nationally is not so different! So by way of digressing, I’d say the tracking of covid-19 vaccinations via surveys is a fantastic example of how polls can be useful if we just lower our silly expectations for hyper-accuracy and laser-like predictive precision.
But back on covid-19 polling… The match between the surveyed self-reported vaccination rate and the official one is soo tight that some pollsters have even started weighting to the figure as a benchmark for survey composition. This clipping shows the weighting scheme for the latest poll of the upcoming Virginia governor’s race by the Washington Post and George Mason University. Note the entry for “Covid-19 vaccination” under the “weighting factors” column, with the benchmark coming from the Virginia Department of health:
I think this is a great effort. In my head, the point of the additional weight is to decrease non-response among culturally/ideologically conservative voters that don’t usually get caught by the traditional demographic weighting schemes — or even those with political weights. Much of the polling error in 2020 came from Republicans who approved of Trump and were very right-leaning culturally not responding to the polls, and therefore skewing the loyalty of the group of “Republicans” as a whole to be too pro-Biden. That error didn’t go away if you simply increased the share of Republicans in the electorate. But maybe it would have been lessened if you could weight by some proxy for underlying social memberships or other conservative attitudes (such as on religion or other non-economic policy issues), especially the conspiratorial ones.
Still, I’m not sure how it will work. One concern is whether partisans are prone to lie about their propensity to get vaccinated at different rates: a sort of Simpson’s paradox where the overall vax rate is right but partisan one is now. For example, the self-reported vaccination rates in YouGov’s data are a bit suspect, as they show decreasing shares of vaccinated people among Republicans in recent weeks. That could be indicative of expressive responding, where people who identify as Republicans want to seem even more Republicans and so say they haven’t been vaccinated. We don’t have evidence for this, but it’s something to keep your eye on.
Polls also look to be overestimating the share of vaccinated young people, and underestimating the share of vaccinated seniors. So while weighting to self-reported vaccination status might get the overall population breakdown right it would not fix any underlying ideological/cultural/etc differences — unless pollsters adjusted on those age-group breakdowns too. Luckily those estimates are also publicized by the CDC, so that’s an attainable variable to add to a weighting scheme!
(NB: But then you run into dangers of over-fitting. There are always trade-offs with these things…)
But all this weighting talk is secondary. The upshot, for now, is that self-reported vaccination rates give analysts an opportunity to compare pollsters’ data to a high-quality government benchmark of political behavior in America. So far, they seem to be passing that test. The polls each week have been well inside the margin of sampling error with the raw CDC rate, even though there are some errors under the hood.
But if you think, as I do, that polls are a crucial tool for increasing the power of the people in Washington, this is a welcome and substantive victory. Three cheers for the polls! (For now…)