If you want your conservatism taken seriously YES.
Funny... I can remember the time when the same thing was said, except with "Republican" replacing "Tea Party."
Slavish devotion to any "movement" or "party" is what got us into this mess in the first place. No party is the answer; conservatism is. If your devotion is to conservatism then your party is simply a vehicle. If your devotion is to your party, then your conservatism is a vehicle... to be abandoned when convenient.
So who cares if he's Tea Party? Is he conservative...?