2020 polling part 2: Incompetence or partisanship? Pollsters need to explain their failure…again

If you missed part 1, in which I highlighted some of the worst polling we saw in 2020, it’s here.

Now let’s move on to something which I don’t think is terribly controversial at this point. Having done this 2 elections in a row, the media’s credibility is pretty well shot. You don’t have to take my word for it. Plenty of observers on the left are saying it right now. Writing at the Atlantic, David Graham has a piece whose headline calls it a catastrophe for American democracy:

This is a disaster for the polling industry and for media outlets and analysts that package and interpret the polls for public consumption, such as FiveThirtyEight, The New York Times’ Upshot, and The Economist’s election unit. They now face serious existential questions. But the greatest problem posed by the polling crisis is not in the presidential election, where the snapshots provided by polling are ultimately measured against an actual tally of votes: As the political cliché goes, the only poll that matters is on Election Day. The real catastrophe is that the failure of the polls leaves Americans with no reliable way to understand what we as a people think outside of elections—which in turn threatens our ability to make choices, or to cohere as a nation.

In short, there’s no reason anyone should trust the polls or the media that relies on them at this point. NPR’s media analyst David Folkenflik reached a similar conclusion:

Hard not to conclude that it’s another black eye for polling and for the news organizations that rely on them. And why do I say that? Well, most of these polls, you read them in newspapers, you see them on TV, you might hear them on our network. They have media branding on them, even those done by colleges or private firms. The polls and the medias are intertwined on this stuff. So when you see a lot of wrong results, that reflects on us, too, regardless of whether Biden or Trump ultimately wins. And it looks like Biden – things are going in his direction. That huge blue wave, I don’t think we’re seeing it. The Dems were projected pick up seats in the House. They’re losing seats in the House…

People thought 2016 was a hot mess, and this may end up being worse for polling.

I like the bit about polls and the media being intertwined. Another way to put it is that pollsters produce a product and the media distributes that product. In fact, in many cases, the media is directly funding the product.

But it’s worth noting that early on election day David Folkenflik published a story at NPR headlined “Down In The Polls, Trump Seeks Familiar Embrace of Conservative Media.” Maybe David needs to do some reflection of his own on the ways in which polling and his own work are intertwined.

Finally, even at Jacobin, a socialist/communist online magazine, the conclusion is that “polling is dead.”

After the debacle of the 2016 election — when polls were off by an average of 5 points — polling institutions revised their methods. Some started dialing more cell phones. Others shifted focus toward swing states. Many began weighting responses by education. “Perhaps, after four years of hand-wringing, the polls will show they were all right after all,” FiveThirtyEight surmised after conducting its own survey of leading pollsters less than a month before the 2020 general election.

So far, the error rate looks the same. Bookmakers, looking at the polls, put their money on Democrats taking back the Senate. That now seems vanishingly unlikely. Polling outfits divined a blue wave that would expand the Democratic majority in the House. The Democrats lost seats.

If 2016 didn’t prove it, 2020 certainly did: the polling industry is in crisis.

This will make the media’s pre-existing credibility problem worse

Let’s not dance around the elephant in the room here. Trust in the media is largely divided by political party. Republicans have always trusted the media less than Democrats (and for good reason) but GOP trust in the media took a steep dive in 2016. This polling is from Gallup (yes, I’m relying on polling to criticize polling, but what else can you do):

Note also that Democrats trust in the media seemed to take off shortly thereafter. Twenty years ago you had 5-15 point media trust gap. Since 2016 you have a 50 point trust gap. There is now a pro-media party (70+%) and an anti-media party (10-20%). And all of that was true before this election where polling was once again badly off both nationally, in critical swing states and in important Senate elections. Not all of those errors favored Democrats but most of them did. So there’s every reason to think trust in the media will decline further, especially on the right.

Why did this happen…again?

I’ve seen various attempts to explain how the media-polling industrial complex got this wrong for a second election in a row. One leading theory is that “shy” Trump voters are not sharing their real preferences with pollsters. Does that mean respondents are lying? Not necessarily. As Jacobin points out, people don’t have to lie when they can simply chose not to answer the phone. In fact, that’s increasingly what happens:

In 2019, Pew Research reported a continuation of the long drop in response rates to a new all-time low: six responses in every one hundred contacts. Commercial pollsters are thought to receive even lower response rates, but they decline to publish that figure. The sampling pool for opinion polling techniques has become so small that it fundamentally skews the data they return. Surveyors know this well and are doing what they can to compensate, but the truth is that random selection polling methods have been obsolescing for decades.

Some more transparency about the actual process might help. Is it really true that pollsters are calling 20 people to get one response? And if so, doesn’t that suggest a kind of self-selection among respondents (most people can see who is calling) that renders the polls suspect.

A Washington Post piece on polling suggests that inconsistent response rates force pollsters to make a lot of guesses about the shape of the electorate. And those guesses may be as significant as the data they actually collect:

The Democratic consultant who spoke on the condition of anonymity said that pollsters, both inside and outside of campaign efforts, have been forced to increasingly make estimations because of an inability to get consistent responses across all voter groups.

“Where polling is fundamentally broken now is these guys have to guess at what the electorate will look like and sometimes guess at how the electorate feels,” this person said. “So we are piling guesses upon guesses upon ­guesses.”

Is this incompetence or partisanship?

I’ve seen and heard plenty of people from the president on down suggesting these are “suppression polls,” i.e. that the pollsters are intentionally shifting things in such a way that Democrats hear what they want. I don’t think that kind of intentional malfeasance can be ruled out because, frankly, we see it happen in the rest of the media as well. What makes pollsters any different?

In any case, the only people who could potentially rule out partisanship are the pollsters themselves and why would anyone believe them? They have an obvious vested interest in protecting their brands. But whether we believe them or not, they owe us an explanation for this failure.

If polling, collectively, were any other industry, they would have gone out of business by now. If TV manufacturers got the size of the screens wrong by 5-10% (is this 60″ TV actually a 54″ TV when you unbox it?), people would bring their own measuring tapes to Best Buy. If a carpenter’s measurements were often 5% off and your living room turned out to have 7 1/2 foot ceilings instead of the standard 8 foot ceilings, you’d demand a repair or a refund. If your lawyer suddenly announced your share of the settlement would be 10% less than he’d said previously, you’d demand to know why and maybe think about reporting him.

Professionals in every industry are expected to be more reliable than this and to explain when they get it wrong. So if these pollsters are professionals (not partisans actively working toward an outcome), where is the public explanation of how they got it wrong? After all, we’re not talking about a TV or a settlement here, we’re talking about an industry that plays a key role in our entire democracy.

To the extent the pollsters at Quinnipiac or some of the others who were badly off won’t explain precisely what went wrong, people should be able to infer certain things from that. When a man on trial for a crime won’t take the stand or pleads the 5th rather than face questions, there’s probably a reason. It suggests, in this case, that partisanship may have invaded the process in some way and no one wants to admit it. There’s a clear partisan bent to the polling. To restore any confidence the industry needs to offer a convincing non-partisan explanation.

And along with that, how about an apology to all of the Americans around the country who, intentionally or not, were badly misled by this industry’s product? That will be subject of part 3 of this series.

Update: 2020 Polling Part 3: Bad Polling Shaped The Election

Continue reading at Hot Air