I think the people who are leaving the state are young families with children, people who can't afford homes in this state, and that is a conservative constituency.
It's been pointed out time and again that Massachusetts transplants to New Hampshire are more reliably Republican than native New Hampshirites, many of whom have shifted their voting habits, but it seems like N.H. people still look to us to blame for their state's turning a little more purple.
Actually, my wife and I are almost 50 with no children. The other people we know who are moving are generally in our same general demographic situation. But, overall, you might be right. People are leaving for a variety of reasons, though.