« Quick Links | Main | "The Democrat Party" »

November 22, 2006

Senate Polls: Democrats Beat the Numbers

By Neil the Ethical Werewolf

Before elections, there's always a lot of fussing about whether the polls are accurate, and how they might be misrepresenting the situation.  Afterwards, when all the data is available, the election is over and nobody really cares what the polls said.  But in deference to my past self who was obsessing over the polls, I'm looking at this summary of how each pollster did on the Senate races they covered.  The following numbers represent the extent to which each pollster overestimated Republican performance on average, and the average error.  As it turns out, each pollster generally overestimated Republican performance. 

Rasmussen: 2.15% GOP overestimation, 3.23% average error
Mason-Dixon:  3.73% GOP overestimation, 3.73% average error
Reuters/Zogby: 1.67% GOP overestimation, 4.56% average error
Zogby Battlegrounds:  5.36% GOP overestimation, 6.45% average error
SurveyUSA: 2.55% GOP overestimation, 5.22% average error
Quinnipiac: 3.67% GOP overestimation, 3.67% average error (only 3 polls)

The formula for GOP overestimation is: ((GOP score in all polls - Dem score in all polls) - (GOP score in all elections - Dem score in all elections)) / number of races sampled.  For the average error, I just averaged how much all of a pollster's final polls missed the margin by.

As far as error goes, I'm not surprised to see that the Zogby Battleground survey, which used some kind of online sampling method, had the most error.  I've never understood how you could do anything online that would give you a reliable sample. 

I'm curious about why all the polls overestimated how well Republicans would do.  One possibility is that Democrats were even more motivated than anybody expected, and that their turnout exceeded what the likely voter screens suggested.  Our get-out-the-vote efforts, powered by enthusiastic Democratic volunteers, may have played a role in that.  There's also the possibility that this was the election when the inability of pollsters to call cell phones finally had a big impact.  I and a lot of young people like me have cell phones but no landlines, so we're undersampled by phone polls, which can't call cell phones.  If you're a pollster, you'll probably want to suggest that the polls were accurate and the races just broke towards the Democrats at the very end. 

November 22, 2006 | Permalink


"If you're a pollster, you'll probably want to suggest that the polls were accurate and the races just broke towards the Democrats at the very end."

Well, given that the exit poll showed that folks who decided in the final 3 days overwhelmingly broke Democratic, isn't this the obviously correct hypothesis?

If a poll showed a race 3 days out as 49D, 49R, and 2UND, and if the race ended up being a 51-49 Democratic win, that poll was probably perfectly accurate when taken, even though the WSJ methodology would (incorrectly) show that poll as overestimating GOP support by 2 points.

Polling is a snapshot of the present, not a prediction of the future.

Posted by: Petey | Nov 22, 2006 7:11:42 AM

As to cell phones, via a Mark Blumenthal mention, here's some evidence strongly implying that you and Karl Rove are both wrong. The headline of the Pew Poll is Cell-Only Voters Not Very Different.

Posted by: Petey | Nov 22, 2006 7:20:45 AM

The breakdown of late deciders was 57-39 in favor of the Democrats. That seems like it could account for a fairly large portion of the gap -- at least, in a race with lots of undecideds.

I suppose it'd be good to see whether the races with the biggest GOP overestimations were the ones with the most undecided voters.

Posted by: Neil the Ethical Werewolf | Nov 22, 2006 7:20:59 AM

Despite the headline on that cell survey, Petey, it seems to suggest that landline people and cell people are different -- or different enough for these purposes. We have a 13% advantage with landliners and a 20% advantage with cell-only people. Sure, it can be swallowed up when we round off in the overall data, but the increments we're trying to explain here are small enough that half-percent differences go a big way.

Posted by: Neil the Ethical Werewolf | Nov 22, 2006 7:35:33 AM

"it seems to suggest that landline people and cell people are different"

Of course, one of the primary ways cell people are different from landline people is that cell people don't vote. Seriously, I'm not sure there is any grouping you could come up with that would be less likely to vote than cell people other than homeless people.

The past 3 elections have all been marked by the side that was behind in the polls arguing that cell people were going to save them. All the data I've seen seem to indicate it's a lousy argument.

Posted by: Petey | Nov 22, 2006 8:04:43 AM

According to the data, 6% of registered voters are cell-only. We have a 7% advantage among these people. Multiply, and you get a 0.42% advantage. Which isn't a lot for other purposes, but when you're trying to cobble together an explanation of a 2-3% discrepancy, it's not negligible. (An 18% advantage with late-deciders in a race with 5% undecided counts for an 0.9% advantage. Which is bigger, but not dramatically so.)

Of course, I'm probably trampling over a lot of margin of error / statistical significance issues here. But if we pay attention to those, we're left with a situation where we basically have no data, and all we can do is cook up intuitively plausible theories.

Posted by: Neil the Ethical Werewolf | Nov 22, 2006 8:18:57 AM

"Multiply, and you get a 0.42% advantage."

OK. But guesstimate a 1/2 ratio of likely voterhood between cell/landline people, and now were down to a 0.21% advantage.

"we're left with a situation where we basically have no data, and all we can do is cook up intuitively plausible theories."

Quite true.

My educated guess is just to discount any thing really being here.

In '06, the polls leaned (R) compared to the final results. In '04 the polls were about even compared to the final results. In '02 the polls leaned (D) compared to the final results. All three of these effects are quite nicely explained by where the late deciders went, as shown by the exit polls.

So I'm pretty comfortable with the late decider hypothesis as the primary cause of partisan poll bias. Cell phone issues still seem like rounding errors to me.

Posted by: Petey | Nov 22, 2006 8:55:41 AM

Wulsin conceded. I blame you for not trying harder, Neil.

Posted by: Petey | Nov 22, 2006 9:44:28 AM

"Wulsin conceded."

It's interesting. After the HAVA voting legislation, in both '04 and '06, a lot of folks including me have been waiting for an election where the (R) won the initial count, but the (D) won with the provisional ballots - the theory being that provision ballots should skew very heavy (D). But it hasn't happened that way in a single election so far.

Posted by: Petey | Nov 22, 2006 10:48:33 AM

Oh Werewolf beware, by thinking cell-phones skew polls you are putting yourself in the same camp with Karl Rove who used skepticism to ignore all the poll results. One of the points of the cell phone thing is that controlling for demographics, cell phone users aren't much different than non cells, and most good pollsters balance for demographics (though not the most prolific).

Also, your Senate sample may just suffer from disparaties among races. Republicans running in Democratic states way underperformed the polls, like to 10 points, but in other states it was more reasonable, and in more conservative states (VA, MT) the Democrats slightly underperformed. Which leads to the conclusion that polls didn't value party loyalty enough, which is what the entire "we should balance for party ID, no we shouldn't, is about".

Lastly, yes Neil, I blame you personally for Wulsin's concession.

Posted by: Tony v | Nov 22, 2006 11:50:30 AM

"Lastly, yes Neil, I blame you personally for Wulsin's concession."

If only Neil could have knocked on a few more doors. If only Neil could have dialed a few more union voters. If only Neil could have videotaped Jean Schmidt insulting him, we'd have pulled out this race.

And if we'd had a good night in the Ohio House races, rather than having a miserable night in Ohio, we'd have made it up to 236 House seats, and I'd have won enough kitty at TradeSports to buy a fucking yacht.

I blame you, Neil. You owe me money.

Posted by: Petey | Nov 22, 2006 12:07:06 PM

One mustn't just extrapolate directly from the margin a poll shows to the margin you expect there to be on election day -- a smart pollster will tell you that any number of other factors need to be taken into account. In particular, complementing the exit polls' evidence about late deciders, there's a well-known phenomenon in polling that undecideds tend to break away from the incumbent toward the challenger. Hence, when most polling is being done in districts with incumbents from the same party (this year, Republicans), you would expect the polls on average to overestimate their support.

Posted by: Taren | Nov 22, 2006 1:35:36 PM

I wonder how many people publically refused to stop supporting Republicans but knew in their heart of hearts that the instanity must be stopped.

Like Fred Jones! I bet Fred Jones caved and voted a straight D ticket once he was in the booth. Then went home and cried like a little girl. And I bet he'll deny all of it.

Posted by: Hamilton Lovecraft | Nov 22, 2006 1:52:19 PM

"In particular, complementing the exit polls' evidence about late deciders, there's a well-known phenomenon in polling that undecideds tend to break away from the incumbent toward the challenger. Hence, when most polling is being done in districts with incumbents from the same party (this year, Republicans), you would expect the polls on average to overestimate their support."

The 'incumbent rule' is just a subset of the larger 'late decider' phenomenon. You can often use the incumbent rule to try to predict which way the late deciders will go.

But it's worth noting that the incumbent rule has not been very reliable at all over the past decade, although it did work perfectly in the '06 cycle.

Posted by: Petey | Nov 22, 2006 1:54:17 PM

Online question for you, Professor McCormick. Paul from Minneapolis asks, "How important is is that many young Howard Dean supporters do not have landline telephones and are not counted in the polls?"

McCormick: Well, I think it's very important, because the kind of polling that's done - and we did one here at - in our department, the Iowa State Caucus Poll - we were using landline phones only, and so we missed some people. Now, the key, of course, is that these younger voters who only have cell phones actually register, and register in the right precinct, and then attend the caucuses. I mean, that's sort of the other part of the equation that should be kept in mind.

--NPR Iowa Caucus coverage, 11a.m., January 19, 2004.

Posted by: Chris | Nov 22, 2006 3:02:32 PM

Alas, I am one werewolf and not a thousand, with $300 and not $30 million to spend. Were I otherwise, this would be a happier day.

Posted by: Neil the Ethical Werewolf | Nov 22, 2006 3:20:50 PM

If it had been a full moon you could have done it, Neil.

Posted by: Sanpete | Nov 22, 2006 3:27:42 PM

Until we actually start including cell-only voters in political polls, we won't really know the effects of having excluded them. There was plenty of underperforming by Republicans, but also plenty of reasons:

-Undecideds tend to break against the incumbent, especially when the media narrative appears to be wholly out of the incumbent's hands.
-People also like voting for winners. If immediately before voting, they're pretty sure one side is going to win, they're more likely to vote for that side, even if they've publically supported the other.
-As an addendum to the above, if Republicans have a feeling that it's going to be a bad year for Republicans, they aren't going to go out and vote. There are exceptions, by which I mean people who think they'll have to answer to skipping an election when they get to Heaven, but this is a phenomenon that does apply to both parties.
-This election marked a shift in the electorate (yes, like '94, or '74.) Polling breakdowns probably skewed more conservative than they should have, all things considered.

Saying cell phones are ineffectual or unlikely to go to the polls is a strong claim to make without evidence. However, the evidence isn't conclusive on the differences between cell-only and landline voters.

Posted by: Jon O. | Nov 22, 2006 7:18:20 PM

"Alas, I am one werewolf and not a thousand"

A feeble excuse.

The date of this election had been known for quite a while. If you had planned ahead and spent some time on the moors of Ohio this spring, you could've bitten a thousand Ohioans to turn them into a thousand werewolves in time for the election.

Your lack of lycanthropic foresight is hurting America.

Posted by: Petey | Nov 23, 2006 12:22:06 AM

FWIW, it seems I was wrong earlier. Wulsin has NOT conceded. The AP called the race for Schmidt, but Wulsin is waiting until all the provisionals are counted. I got the idea that Wulsin had conceded from TAPPED, but they were wrong.

Posted by: Petey | Nov 23, 2006 1:06:45 AM

Of course, in the realm of solutions that are actually within somebody's power, John Lapp could have taken a look at the public polling and put a small amount of money into what was a very cheap race.

Posted by: Neil the Ethical Werewolf | Nov 23, 2006 5:49:27 AM

The aged people who have lost their virility after falling in the grip of erectile dysfunction can gain it back through levitra. This anti-impotency cure is definitely cheaper than all other erectile dysfunction medicines discovered so far and as such been termed as cheap levitra.You can order levitra from the comfort of your home through levitra online. After the drug Levitra is administered to the penis, it increases the flow of blood to the penis and consequently the penis becomes engorged with blood and sets off erections necessary for satisfactory physical intercourse. Levitra has a proven track record of curing erectile dysfunction in people across the world and so you can uninhibitedly buy levitra. Buy Levitra online and make your way out of male impotency.

Posted by: levitra | Dec 17, 2006 10:58:08 PM

Obesity has become one of the major issues in US nowadays. Over 50% of the population is suffering from obesity & look for easy solutions to weight loss at any cost. But you need not to pay many dollars for losing weight. You can simply buy adipex which has proved as an effective diet pill in recent years & benefited millions of people. adipex simply reduce your hunger cravings & gives you full feelings. This prevents you from eating extra calories that add up fats on your body. You can get this magical diet pill just by one click as online adipex is available at your ease. What you are thinking now? Just order adipex now & get an attractive shape with great health.

Posted by: adipex | Jan 15, 2007 3:53:54 AM

debt consolidation Today student loan debt consolidation is a big debt free issue in the United States. consolidating debt Most students in the US pursue debt consolidation loan further studies through external finance. bad credit debtThe government debt free too pumps in millions of dollars into the economy to finance these studies and provide a congenial environment for proper and high class study to the students. cheap debt consolidationEvery student has a different set of financial requirement. There are several mortgage refinance organizations and student loan best debt consolidation consolidation companies, which provide student debt consolidation. debt consolidation help There are several programs for credit card debt individual needs debt help of the student. The result of all the programs is similar. All the programs work debt consolidation help towards designing debt consolidation online a payment plan suiting your individual needs in which the program will merge all your student loan existing loans into a single consolidate debt sum and reduce your existing interest rate in an effort to bring the repayment debt consolidation help monthly installment to a sum debt consolidation solution which you can pay easily with your existing income.debt plan

credit card debt

Posted by: debt consolidation | Aug 5, 2007 5:17:55 AM

As the Pfizer manufactured erectile dysfunction drug Viagra creates record in drug sales throughout the world after men in hordes opted to buy cheap viagra through viagra online, other rival drug manufacturing companies have lost their peace of mind. They are consistently trying to diminish the ever-growing Viagra popularity by spreading that excessive usage of Viagra has accelerated the prevalence of HIV and other sexually transmitted diseases among men. Of, course it is true men taking Viagra for recreational means are vulnerable to sexually transmitted diseases and as such if a man chooses to buy viagra from any online drug store without a Viagra prescription from the doctor, his health is totally at risk. The4proper way to combat erectile dysfunction with Viagra is to buy viagra online from an authentic drug store and administer the drug in accordance with the suggestions of the doctor.

Posted by: viagra | Aug 14, 2007 10:10:10 AM

The comments to this entry are closed.