Skip to content, or skip to search.

Skip to content, or skip to search.

‘‘The. Polls. Have. Stopped. Making. Any. Sense.’’


Today, some demographers think that perhaps a majority of households either don’t have or don’t answer a landline. In other words, ignoring cell phones risks ignoring more than half of America. “I don’t know how you can in good conscience release polls in this day and age without that group factored in,” says Leve, who, in addition to robo-­polling machines, now spends money on live operators so that he can reach cell-phone users, too. Survey USA, however, is the exception: Especially when it comes to state-level polls, many firms, including PPP and Rasmussen, continue to exclude cell phones in their surveys.

But let’s say you’re a polling firm that has decided to spend money on cell phones. You still need to confront the bigger question: What to make of your results? No firm publishes its results without some sort of adjustment, but nobody agrees on what the proper adjustment should be. Should the cell-phone respondents represent 25 or 30 or 40 percent of a pollster’s sample? Considering that response rates for cell phones are even lower than those for landlines, at what point do the weights applied to cell-phone users, in order to get them to account for, say, 35 percent of a poll’s sample, produce unstable results? “You could ask a lot of different people, ‘Well, how do you combine your cell phones and your landlines?’ and the answer is, ‘Oh, it’s a real delicate art,’ ” says Leve. “No one knows the right way to do this right now.” “It’s almost a miracle that this stuff is still projective,” says Blumenthal. “The whole idea that we know the probability of selection for every adult is a little bit of a fiction.”

Most pollsters weight their responses using some combination of data from the census and previous elections’ exit polls—the latter of which, of course, are polls themselves, and which require pollsters to quantify their speculations about how the current political climate compares to the last one. A few firms, most notably Rasmussen, weight by party identification. (That’s why Rasmussen, which currently structures its polls so that Republicans account for 37.6 of respondents and Democrats for 33.3 percent, is now alone among pollsters in showing a tight presidential race.) And some pollsters just basically wing it, weighting their polls based on their hunches, like the pollster who decided that even though black turnout in Michigan has historically been between 12 and 14 percent, it’ll be 8 percent in 2012—an assumption that, in an August poll, tied Romney with Obama, against all other evidence.

Some pollsters are giving up on weighting—and telephones—altogether. Doug Rivers, a Stanford political scientist who runs the Internet polling firm YouGov, is leading the way in that area. Working from high-quality government surveys as well as data from Pew, Rivers has created a multi-variable sample of the population. At the same time, YouGov has recruited an online panel of a million people who, in exchange for a fee, agree to respond to YouGov surveys. From that panel of a million, YouGov then selects a subset of 1,000 to 2,000 that matches the variables of its population sample and has that subset take its survey. “Our method is purposive selection as opposed to random selection,” Rivers explains. “So if the variables we’ve used to select people don’t remove the selection bias for joining the panel, the biases will follow.” But Rivers is confident that he’s got the right variables.

Still, despite a solid track record in the 2010 midterms and the GOP presidential primaries this year, YouGov has its doubters. The Real Clear Politics Poll Average, for instance, excludes YouGov from its results. Which is one reason Rivers is already looking beyond online panels as the future of polling. During the Republican National Convention in August, YouGov launched a pilot project with Microsoft that sought the political views of Xbox Live users—asking them to watch the proceedings and convey their responses through their gaming systems. It plans to do the same thing during the three presidential debates starting this week, producing real-time polls. “CNN used to put ten people in a room during a debate and gave them dials to express how they were feeling about what they were watching,” says David Rothschild, a Microsoft economist working on the Xbox project. “But we could have hundreds of thousands of people with controllers in their hands.” Although those hundreds of thousands of gamers almost undoubtedly skew young and male (the exact opposite of telephone surveys), YouGov and Microsoft believe that with the proper weighting, they’ll be able to produce that mythical representative sample.“If George Gallup was frozen in time after the 1936 election and came back in 2012, he’d recognize polling as exactly what he’s doing, but that’s going to be revolutionized in the next few years,” predicts Rothschild. “We’ll be thinking about data coming out in real time.” In other words, if you think the torrent of polls and numbers is overwhelming now, just wait.


Current Issue
Subscribe to New York

Give a Gift