Are you serious? Though he didn't mean it that way, what Romney said was interpreted as class politics. It painted him into the picture the Democrats framed of an aloof, uncaring rich guy.
Some conservative voters assumed Romney wasn't for them and didn't go to the polls. Call them Huckabee voters. I don't know how many of them there were, but the reaction of some of them was to assume Romney was talking about those who had less money than he did.
That may have been the wrong interpretation and the wrong reaction, but it was to be expected. That's why candidates don't talk that way when they know the cameras are on them. The upscale voters who'd gone for Obama in 2008 that Romney counted on winning based on his economic abilities and their economic interest didn't go for him either, so he lost.
Difference is, I wouldn’t call Huckabee voters conservatives. Idealogical, single issue, social conservatives maybe, but not simply conservatives. While this doesn’t seem to address your whole post, it actually does - and it’s not necessarily a bad thing, it just simply is.