At the very least, a mid-term party switch should trigger a special election back home, to insure that your constituents approve of your move. IIRC, Phil Gramm did this when he switched to the GOP. The time and effort involved in such a process might serve as an effective deterrent to party-switching.
The thing that rankled me the most about Jeffords' switch was that less than half a year earlier he used GOP money and resources to fund his own re-election, and perpetrated fraud upon his electorate by running as a Republican.