"And since the federal government is NOT empowered to restrict these things, then those powers are reserved to the states, or to the people."
Only if the State Constitutions specifically empower the states to interfere. If the state constitutions don't mention it, they cannot make up a government power out of thin air. Our rights do not need to be enumerated, but a government power DOES have to be specified by either the Federal or a State Constitution.
Even if a Constitution did include a clause that would allow the government to interfere with an activity that did not infringe on somebody else's rights to life, liberty, or property, that would just make that Constitutional clause tyrannical. There is never an excuse to make such a clause.
I realize that your libertarian philosophy is summed up by this idea, but it's not to be found in the Constitution of these United States.