How much can we trust these polls out of Georgia?
There is more uncertainty than even aggregators are letting on
I want to think about these Georgia polling averages some more. Can we really be confident in the movement they purport to detect? Is this really the signal in the noise — or is it just noise?
Editor’s Note: This is a paid post for premium subscribers. If you are a subscriber and have friends or family that you think might learn something from this post, you should feel open to forward it to them regardless of their membership status. But if you have been graciously sent this post by a friend that subscribes, please consider signing up for updates yourself by hitting the button below.
FiveThirtyEight, whose averages are generally the best in the business (and also the only ones for the runoffs), says that support for the Democratic candidates in Georgia increased by about one-to-two percentage points between November 9th and December 10th, 2020, then decreased by one point over the next two weeks, and then went up by two points again between Christmas and today, January 4th. My question is: is this movement real? Or is the average just painting a mirage of a dynamic race?
Clearly, if you squint, there are some dynamics to the contest. Maybe Democrats gained ground while Trump was crying foul with the election, then lost is, then gained more as Mith McConnell and some Republicans blocked $2,000 stimulus checks. But is the squinting right?
Think about it like a regression, where the average over time is the line of best fit and each poll is a point on the graph. My guess is that the residual error between each poll and a straight line, representing a constant mode, would not be larger than the residual error for the average that 538 has calculated to a statistically significant degree. And if that's true, how can we be sure that the averages aren't just showing noise?
Maybe it would be better to show the polls also as a histogram, distributed around the global mean. That might give readers a good understanding of whether the polls have really changed.
When we're talking about differences of one or two percentage points when the true margin of error for a single poll is close to seven points for a single percentage (and 13-14 points for the differences between candidates), this whole "let's show the average as evolving and ignore the margin of error" business can be pretty misleading.
My advice to you all is to take the polls with a grain of salt. I bet I could convince you that either the Democrat or Republican is ahead if I relied on choice data sources. That should tell you all you need to know about how confident we can be about either side.
Tomorrow's race in Georgia is a true toss-up, plain and simple. Let's leave it at that and wait for results to come in. I’ll probably be blogging about them tomorrow.