There has always been a limit on how people can live, regardless of what they "think is right." Obviously. And no one individual has the freedom to force chains on anyone.
People who freely chose slavery are not free anymore, especially that if many such choices are made the process is irreversible.
I understand your point. The people give up some of their rights and then it's awfully hard to get them back. I hear you. We have always been free to give our rights away. I think people have gotten better at it over time.
You can give your own rights away. Other's rights are not yours to give.