I am not confusing these, at all. Much of the unbridled liberty that you lament as leaving is simply going away due to our growth as a nation.. You can eat any amount you want if you live out in the wild west not surrounded by neighbors. Once the city grows up around you and you join an insurance pool (health ins.), your diet does adversely affect my premiums. Years ago you could shoot a gun in your backyard, but now it endangers my kids. But, you cry, no one should be able to limit your right to smoke, shoot, eat, drive, etc. Of course not... if only you are affected by that etc. But, the larger we get, the closer we get and more personal liberty will get constrained. So, the infringement that you see declining is partially by necessity, so that you don’t “...cause harm to others.”
The more people there are, especially in close proximity to one another, the more restrictive the environment becomes.
People generally don't like each other very much, and if you're doing something I don't like (whether or not it hurts me) I will try to put a stop to it, sometimes just out of spite. I can make a case for almost anything you do being harmful to me. Multiple this sentiment by a few hundred million and you're well on your way to oppression.
If people were truly rational we wouldn't be having this discussion but unfortunately emotions get in the way.