Sam Stein: New Hampshire Polling Fiasco Not Unique In History
[Sam Stein is a Political Reporter at the Huffington Post, based in Washington, D.C. Previously he has worked for Newsweek magazine, the New York Daily News and the investigative journalism group Center for Public Integrity.]
So the polling industry, and the reporters who religiously follow it, got their predictions wrong. What was expected to be an easy victory for Sen. Barack Obama in the New Hampshire primary turned instead into a dramatic win for Sen. Hillary Clinton.
But if the failures of predictive surveying and journalists who buy into it came as a shock, it shouldn't have. Long before the unforeseen results of the 2008 New Hampshire Democratic primary, pollsters had numerous shortcomings just as glaring.
A month before the 1996 presidential campaign, Bob Dole went from trailing Bill Clinton by 17 points to being just 2 points behind a week later. In the end, however, he ended up losing by 9 points.
Four years later, a similar occurrence. In the weeks before voting took place in the 2000 election, CNN sponsored two separate polls, with two very different findings, both released on the same day. The first, a CNN/USA Today/Galllup poll, showed George Bush 13 percentage points ahead of Al Gore - 52 to 39 percent. Two hours later, CNN/Time released a poll showing Bush's lead at 6 percent. Keating Holland, the station's polling director said, at the time, the two findings were "statistically in agreement."
In mid-September 2004, a Gallup poll found Sen. John Kerry trailing Bush by 14 percentage points. But the poll, it turned out, was greatly weighted to GOP voters, assuming that 40 percent of voters on election day would be Republicans and only 33 percent Democrats. Historically, this was off. In the previous two presidential cycles, Democrats accounted for 39 percent of voters.
The same inaccuracies have been seen in non-presidential races. In November 1998, several pre-election surveys proved drastically off-the-mark. Ten days before the election, Wisconsin Sen. Russ Feingold was tracked as losing to Rep. Mark Neumann, 43 percent to 46 percent. Feingold wound up winning by two percent. That same year, in the Minnesota gubernatorial race, Hubert Humphrey III was trouncing Jesse Ventura 35 to 27 percent just three days before the election. "The Body," however, prevailed. And in his victory speech he tellingly declared: "[My fellow governors] can learn from me that the American dream is still alive. They can learn from me: Don't ever believe the polls."
Despite these repeated blunders, pollsters say they have had more successes than misses. In New Hampshire 2008, they point to the GOP race as evidence of getting it right. "Most primary polls have been accurate. This is unusual to have every poll show the same thing and turn out wrong," Frank Newport, Editor in Chief of the Gallup Poll, told the Huffington Post.
Moreover, some of the responsibility rests on the shoulders of the journalists to make sure that the shortcomings of these polls are accurately reported. And indeed, in the wake of the debacle in New Hampshire, political reporters (including those at the Huffington Post) have been forced to engage in a bit of self-flagellation. Writing for Politico John F. Harris and Jim VandeHei today declared:
"New Hampshire sealed it. The winner was Hillary Rodham Clinton, and the loser -- not just of Tuesday's primary but of the 2008 campaign cycle so far -- was us.
"'Us' is the community of reporters, pundits and prognosticators who so confidently -- and so rashly -- stake our reputations on the illusion that we understand politics and have special insight that allows us to predict the behavior of voters."
Of course, explanations are also needed. And following New Hampshire there have been many. White voters, it was theorized, were not honest with pollsters when they said they would support Obama - hoping not to appear as if they wouldn't back an African-American candidate. Independents showed up beyond expected numbers and those who did sided disproportionately and unexpectedly with Sen. John McCain. Clinton's placement on the ballot benefited her more than Obama. Something fraudulent occurred. Democratic pollster Peter Hart, even offered an Occam's Razor-like rationalization: "What it really comes down, in my estimation, is, they stopped polling too soon."
All these explanations, however, do little to prepare reporters to avoid making the same mistakes in the future. And in this regard, experts note, there are several factors that journalists should take into account when using polls.
In New Hampshire, as elsewhere, the response rate to the surveys was extremely low. By one estimate, 80 percent of those called by the pollsters simply did not participate. Who is most likely to do this? Those who either are too busy, too uninterested, have only a cell phone, or are too uninformed. There is no way to be sure how these people would have voted. And pollsters insist that their participation in surveys would not generally affect the results.
"What really matters is, are the people we are not able to complete the interview with substantially different in their political attitudes then those who we do reach," said Newport. "Is there some reason that this small group we have is not representative of the larger group we are trying to estimate? And data shows that the people who don't answer don't seem to be substantially different than those who do."
And yet, it remains significant that a huge portion of the population, for one reason or another, is never weighed into the polls conducted. This group rarely gets mentioned by the press. Undecided voters also receive a disproportionately small amount of scrutiny. According to Jennifer Donahue of the New Hampshire Institute of Politics, nearly half of New Hampshire Democratic voters had yet to choose a candidate just days before the vote. This figure, which is well above the percent of votes received by any candidate, Republican or Democrat, was readily available but routinely ignored.
Reporting on the polls, Donahue told the Huffington Post, should have noted prominently that "half the Democratic electorate is still undecided. The race is still wide open."
Read entire article at Huffington Post (Blog)
So the polling industry, and the reporters who religiously follow it, got their predictions wrong. What was expected to be an easy victory for Sen. Barack Obama in the New Hampshire primary turned instead into a dramatic win for Sen. Hillary Clinton.
But if the failures of predictive surveying and journalists who buy into it came as a shock, it shouldn't have. Long before the unforeseen results of the 2008 New Hampshire Democratic primary, pollsters had numerous shortcomings just as glaring.
A month before the 1996 presidential campaign, Bob Dole went from trailing Bill Clinton by 17 points to being just 2 points behind a week later. In the end, however, he ended up losing by 9 points.
Four years later, a similar occurrence. In the weeks before voting took place in the 2000 election, CNN sponsored two separate polls, with two very different findings, both released on the same day. The first, a CNN/USA Today/Galllup poll, showed George Bush 13 percentage points ahead of Al Gore - 52 to 39 percent. Two hours later, CNN/Time released a poll showing Bush's lead at 6 percent. Keating Holland, the station's polling director said, at the time, the two findings were "statistically in agreement."
In mid-September 2004, a Gallup poll found Sen. John Kerry trailing Bush by 14 percentage points. But the poll, it turned out, was greatly weighted to GOP voters, assuming that 40 percent of voters on election day would be Republicans and only 33 percent Democrats. Historically, this was off. In the previous two presidential cycles, Democrats accounted for 39 percent of voters.
The same inaccuracies have been seen in non-presidential races. In November 1998, several pre-election surveys proved drastically off-the-mark. Ten days before the election, Wisconsin Sen. Russ Feingold was tracked as losing to Rep. Mark Neumann, 43 percent to 46 percent. Feingold wound up winning by two percent. That same year, in the Minnesota gubernatorial race, Hubert Humphrey III was trouncing Jesse Ventura 35 to 27 percent just three days before the election. "The Body," however, prevailed. And in his victory speech he tellingly declared: "[My fellow governors] can learn from me that the American dream is still alive. They can learn from me: Don't ever believe the polls."
Despite these repeated blunders, pollsters say they have had more successes than misses. In New Hampshire 2008, they point to the GOP race as evidence of getting it right. "Most primary polls have been accurate. This is unusual to have every poll show the same thing and turn out wrong," Frank Newport, Editor in Chief of the Gallup Poll, told the Huffington Post.
Moreover, some of the responsibility rests on the shoulders of the journalists to make sure that the shortcomings of these polls are accurately reported. And indeed, in the wake of the debacle in New Hampshire, political reporters (including those at the Huffington Post) have been forced to engage in a bit of self-flagellation. Writing for Politico John F. Harris and Jim VandeHei today declared:
"New Hampshire sealed it. The winner was Hillary Rodham Clinton, and the loser -- not just of Tuesday's primary but of the 2008 campaign cycle so far -- was us.
"'Us' is the community of reporters, pundits and prognosticators who so confidently -- and so rashly -- stake our reputations on the illusion that we understand politics and have special insight that allows us to predict the behavior of voters."
Of course, explanations are also needed. And following New Hampshire there have been many. White voters, it was theorized, were not honest with pollsters when they said they would support Obama - hoping not to appear as if they wouldn't back an African-American candidate. Independents showed up beyond expected numbers and those who did sided disproportionately and unexpectedly with Sen. John McCain. Clinton's placement on the ballot benefited her more than Obama. Something fraudulent occurred. Democratic pollster Peter Hart, even offered an Occam's Razor-like rationalization: "What it really comes down, in my estimation, is, they stopped polling too soon."
All these explanations, however, do little to prepare reporters to avoid making the same mistakes in the future. And in this regard, experts note, there are several factors that journalists should take into account when using polls.
In New Hampshire, as elsewhere, the response rate to the surveys was extremely low. By one estimate, 80 percent of those called by the pollsters simply did not participate. Who is most likely to do this? Those who either are too busy, too uninterested, have only a cell phone, or are too uninformed. There is no way to be sure how these people would have voted. And pollsters insist that their participation in surveys would not generally affect the results.
"What really matters is, are the people we are not able to complete the interview with substantially different in their political attitudes then those who we do reach," said Newport. "Is there some reason that this small group we have is not representative of the larger group we are trying to estimate? And data shows that the people who don't answer don't seem to be substantially different than those who do."
And yet, it remains significant that a huge portion of the population, for one reason or another, is never weighed into the polls conducted. This group rarely gets mentioned by the press. Undecided voters also receive a disproportionately small amount of scrutiny. According to Jennifer Donahue of the New Hampshire Institute of Politics, nearly half of New Hampshire Democratic voters had yet to choose a candidate just days before the vote. This figure, which is well above the percent of votes received by any candidate, Republican or Democrat, was readily available but routinely ignored.
Reporting on the polls, Donahue told the Huffington Post, should have noted prominently that "half the Democratic electorate is still undecided. The race is still wide open."