The Polling Problem

Folks in general and perhaps Americans in particular like scores. We want to know who is winning, who is losing and by how much. In politics, those numbers are provided by pollsters and the polling aggregators. But we have a polling problem.

The political numbers industry is ailing. Its problems distort the electorate’s view of the state of many races.

Way back when, companies would mail, call or even visit folks in the right proportion (age, educational attainment, religion, party, sex, race, etc.) and put the results through formulas – “algorithm” wasn’t yet a popular term outside of math class – to weigh the results. If done correctly, this result would be an accurate assessment of the race. It’s an art and a science.

That system was fine until cell phones blew it up. Separating the voter from his or her physical address was a challenge. A bigger and ongoing challenge is that fewer and fewer people answer calls from unknown numbers. 

Two other parallel transitions occurred. The first is that a group of data professionals emerged that are known as aggregators. Aggregators do not conduct polls. They are statisticians and data analysts who weigh and combine polls to create composite scores. This in theory corrects for undetected bias or other problems with lone polls. It, too, is an art and a science. 

So we have two arts on top of two sciences, so to speak.

The background for all this was an increasing demand for “scores” from a public that was constantly online, accustomed to instant access to data and anxious to know the state of the races that mattered to them.

Can newbies fix the polling problem?

While the explosion of technology made “doing things right” more difficult and expensive, it also made it easier for new pollsters to get into the game. Many of them seek to use the new tools to legitimately assess races. They are not part of the polling problem. 

But things have taken a darker turn more recently. Candidates, like sports teams, thrive on momentum. Being perceived as gaining support during the last days of a campaign is vital. Why not game this rather porous system and create a positive narrative that creates a bit of momentum and a closer race even if the truth is the opposite?

Indeed, there even are cases of completely fake polls. U.S. News & World Report reported on fake polls in California, Wisconsin and Nevada: 

The trajectory of the California poll offers a cautionary tale about the perils of trusting unvetted polling. While high-quality independent polls still exist, they appear amid a slew of other surveys produced by outside groups, campaigns or super PACs or using undisclosed or questionable methodologies. It can be a struggle to make sense of the findings — and as the latest episode shows, even campaigns may end up falling for falsified data.

Complete fakes are the exception, of course. But more subtle hacks are dangerous as well — perhaps more so because they can evade detection. It’s easy: Just tweak the algorithm or ask questions that are designed to elicit a desired answer from those polled. Hopium Chronicle’s Simon Rosenberg and TargetSmart CEO Tom Bonier argue that this is happening. 

The two had a very interesting conversation about it this week. Rosenberg suggests at the end that the Republicans may have given up on the House of Representatives and likely will use this approach in three or four races in an effort to save their Senate majority.

Until this cycle, the response of many aggregators was to include almost all polls and hope that the legitimate ones overcome the insidious impact of the bad ones. This was not unreasonable because the difference between a legitimate and bogus poll may not be black and white. The drama to Rosenberg and Bonier is whether the polling industry will find less passive ways to combat attempts to weaponize their craft. 

The polling problem is a compelling topic to folks who are interested in politics. For those who aren’t, the key takeaway is simple: Polls you’ve never heard of–especially from new or offshore companies or those with cheesy clipart-heavy websites–may not be kosher. Take them with a grain of salt. Also be skeptical of aggregators that include such polls. 

(I had been thinking of writing a polling post for a while. H/T to Rosenberg and Bonier for clarifying my thinking and making my time on the treadmill pass a bit more quickly. Here is a link to the podcast for those who want a deep dive. Image: Leo Reynolds)