Any virtue taken to its limit becomes a vice. That includes freedom.
It’s not a vice until it deprives another person of life, liberty, or property.
The natural limits on freedom are the consequences that arise from exercing that freedom. In the US we have passed on a lot of those personal consequences to the government and society as a whole. Hence the abuse of freedom.
So we don’t have to so much limit freedom as much as letting the person that abuses freedom suffer the consequences.