Happy weekend, subscribers. This is your weekly exclusive post on a notable public opinion poll, new social science study, or related data illuminating something in the news. Feel free to share this post, and please comment if you’d like to discuss it!
On Friday, NPR published an article with a headline reading “Biden gets a bounce after the State of the Union, NPR/PBS NewsHour/Marist poll shows.” On Twitter, they went with “NEW POLL: President Biden saw a significant boost in his approval ratings following his State of the Union address, a new NPR/PBS NewsHour/Marist poll finds” (emphasis is my own).
I think there are a few things just a little wrong with this. But before there, let’s see the data.
The chart shows Biden’s approval rating increasing from 39% to 47% between February 25th, when they published their previous poll, and March 4th when this one came out. Note that February 25th is after Vladimir Putin issued the Russian military to invade Ukraine — that means we can rule out that the bump is a so-called “rally-around-the-flag” effect for Biden as a result of a crisis in foreign affairs. (NB: This is a well-documented pattern in historical approval ratings: see here for more.)
But, actually, the last survey was conducted between February 15th and 21st. That means the opinions people were expressing were formed before Vladimir Putin launched his war, not after. And that means that the bounce in approval ratings could have come from either Biden’s State of the Union address or a potential rally-around-the-flag effect caused by Ukraine. And since studies of historical approval ratings show there is no consistent bump for SOTU, I’m much likelier to believe the case for a rally around Biden.
. . .
Another problem is that NPR is reporting a “significant swing” in approval based only on one poll. But focusing on only two data points tends to exaggerate movement in attitudes above what is really going on. Read the poll’s methodology reports and you’ll find the margins of error for both surveys is about 4 percentage points. Remember that the margin of error applies to both percentages, so the 8 point swing falls just outside the range of deviation we would expect from sampling error alone. (Here is the formula for the margin of error for the difference in proportions.)
What this means is that the real change in approval could be smaller than 8 points, but still greater than zero; perhaps the last poll should have been 41 and this one should be 45 and the actual difference is only 4 points.
But, do we really only expect sampling error here? Remember that the margin of error pollsters report doesn’t take into account the potential for non-response within demographic groups. This means the share of Democratic white voters in the recent poll could be higher than the share in the last poll, and the margin of error doesn’t account for that. Indeed, since Marist (the pollster which fields surveys for NPR) doesn’t weight by education, the share of college-educated white voters (a Democratic-leaning group) increased from 25 to 29% among polled registered voters from the February poll to the March poll. That could artificially inflate any real movement in opinion and wouldn’t be captured in the poll’s margin of sampling error — which means reporters are exaggerating differences in opinions.
. . .
Third, what about the averages? Since we know there is more noise in individual polling than in aggregates, it’s more accurate to look at trends across a host of pollsters. Here, we see evidence for a smaller bounce in Biden’s ratings. Polling from YouGov suggests the bounce could be closer to 3 points. I think that’s a much more reasonable estimate of overall changes in opinion.
. . .
The final point is that it’s still a little early to be looking for a change in Biden’s approval rating as a result of Russia’s invasion of Ukraine. My analysis of past bounces during times of international crisis suggests rally-around-the-flag effects peak at about 6 weeks after the triggering event. Note that the average bounce is about 6-7 points, but they have tended to get smaller as polarization increases.
In the end, I do think Biden islikely to get a non-zero bounce as a result of the crisis in Ukraine. But the analysis of a single new NPR poll is not enough evidence to prove the case.
I agree that it is the “rally-around-the-flag” effect. As you mention, the historical patterns suggest that the State of the Union doesn't really help with Presidential approval ratings. Viewership of the State of the Union has gone down since 1993, so it would be harder for a President to get a bounce assuming there was one.
Hi Elliott,
I agree that it is the “rally-around-the-flag” effect. As you mention, the historical patterns suggest that the State of the Union doesn't really help with Presidential approval ratings. Viewership of the State of the Union has gone down since 1993, so it would be harder for a President to get a bounce assuming there was one.
https://www.statista.com/statistics/252425/state-of-the-union-address-viewer-numbers/
https://www.axios.com/biden-state-of-the-union-tv-ratings-83a01848-9504-4eb6-a449-fe1ef07b4586.html
I hope all is well!
Elliot
Agreed!