Most of those who jumped ship to Obama fall in that "country club" category.
And there is that split between libertarians and social conservatives.
But rank and file support for libertarian ideas ebbs and flows. Or it follows a cycle from a low point, like 1936 or 1964 to a high point like 1984.
When people feel the shoe of government regulation and taxation pinch, they get together to fix things (so long as they believe that that degree of regulation or taxation isn't necessary). When the shoe doesn't pinch, people stop caring about big government. Committed libertarians and small government devotees aren't a large part of the population.
Which I think is mostly the fault of social conservatives. True libertarians at least respect the pro-life view and are willing to tag along to a point. And with regard to gay marriage, no true libertarian would want to expand the role of government, add to the burden of clogged divorce courts, give new Social Security transfer privilages etc.
OTOH, regulating on-line gambling, banning the sale of horse meat etc., regulating campaign speech etc. will drive any sensible libertarian away.