I see your point. Am I free to leave? Yes.
In a constitutional republic, the states could hold different laws, with the fed taking a limited, constitutional role. If that were still the case, we could establish libertarian style communities in the US and I could go to one.
But that's not the case now, is it? Since there are no more 'communities' in the sense of the 9th and 10th Amendments, we would have to say that in practice, the entire US is one big community. And yes, it is surrounded by armed guards.
And yes, if it becomes too restrictive, and fails to observe even the most basic of rights, I will leave.
There are plenty of communities in America, with different laws and practices. While I'd like to see the balance of power shift back more in direction of the states and their political subdivisions, that isn't the goal of Libertarianism. It seeks the abolition of community rights, and the establishment of a borderless world.