Off the bat, his immigration plan (and also his record) is to the left of everyone, from Perry to Newt to Romney. That makes him a nonstarter to most (I'm not to keen on anyones immigrations plans in this race, but going further to the left isn't going to help, if I can't get Tancredo like, I'll take Tancredo lite, or the closest thing I can see).
He also said, and this is a direct quote, he found a "border fence repulsive", there isn't to many ways to spin that. He believes that the country needs more immigration (he actually said this in a debate), things like that add up to the moderate label. Yes, he is good on taxes, and he is pro-life, but he also likes to spend money, believes in climate change (and did commercials for a liberal environmental group), supports gay unions, and is on the left on immigration.
No one is perfect, I respect that, but he is a bit to far away from where I want the candidates to be for me to feel comfortable.
Huntsman may hate the thought of a fence on the border but also believes is a necessity for the country to get a handle on its illegal immigration problem. I don’t know enough about his immigration amnesty program to comment but I also don’t trust the needle threading, flip flopping approach of Gingrich (....and Romney who will change positions within a single debate to whatever gets the most applause).