I don't just hang up. After all, the caller is some minimum-wage peon just trying to make a living. I tell them I never answer telephone polls, politely thank them, and then hang up.
On those rare occasions when they ask me why I don't answer polls, I tell them I'm a professional statistician, and I know how meaningless they are. Which is all true.