Political Polling - Season of the Witch

It's a desire as old as humanity, the wish to know the future. That's why palmistry and tarot card reading are still things even though the only thing palm readers can tell is whether I work with my hands and and tarot decks can only reveal how well I shuffle. Modern, scientific polling is more accurate than reading goat entrails but that is not saying much.

Pollsters will acknowledge that their polls are accurate, within a margin of error, except for those times they aren't. Pollsters like to claim to be masters of an arcane craft or, like Nat Silver, genius statisticians. What they are is educated guessers. Out of over 1,000 polls in the last week before the 2012 election, some pollsters were supernaturally accurate while others missed by a country mile. But, hitting your number in roulette does not make you prescient, just lucky.

When you look inside the magic box of pollsters you find a rather clumsy set of gears.

• Margin of Error -  Assuming all other factors are equal, this is the range that a poll encompasses. A standard 1,000 person survey has a MoE of ±3%, meaning that 19 times out of 20 the actual result will be within that range. So, a poll claiming a 52% win could really be showing a 49% loss. Most of the time the "momentum" polls reveal is nothing more than MoE noise. But, this all assumes all other factors are equal and they never are.
• Survey Structure - I once had a pollster explain to me how he could manipulate the results of a survey just by the order he asked the questions. Professional pollsters are aware of this but not even the best of them can catch all the subtle biases created by survey structure.
• Response Rate - Pollsters are annoying and in this era of cell phones and call screening they are an avoidable annoyance. To get 1,000 responses requires over 20,000 attempts. This lead to....
• Sampling Bias - Polls tend to under sample younger, poorer, and minority people. This means the results are not random. This leads to....
• Weighting - When I was in high school my science teacher had what he call "Stowell's Constant," which was the number you added or subtracted from the result you got to get the correct answer. Pollsters do the same thing. They adjust the raw results they get to mirror what they believe is the electorate. Rasmussen Reports notoriously believes there are more Republicans that actually exist and their results tend to skew Republican.
• Weighting Small Samples -  A problem with weighing is when you start parsing polls you get some really small sample sizes. A standard 1,000 person survey would expect 70 African-American males. If the poll only actually reached 30 black men then the MoE is approaching 18%, an error that is magnified by applying weighting adjustments. Pollsters hope the accumulations of these small sample errors will cancel each other out but there is no science to that, it is just hope.
• In Hindsight - Pollsters always come up with explanations for when they screw the pooch but whenever they get close to the mark it is because of their mastery of the art of polling.

I'm not saying that political polls are worthless. They are amusing and, if taken with an appropriate number of grains of salt, mildly illuminating. They can tell you if an election will be close, but so can common sense. Polls are most useful for people who lack any common sense which is why politicians cling to them so desperately.