I agree with you that the federal government has no real constitutional authority over these areas, but we weren't really arguing Federalism. You would agree, would you not, that the states DO have these powers, if they so choose? Yet I would guess that Libertarians would argue against that authority as well.
I am a small government conservative, but that does not mean that I am in favor of anarchy. If men were angels (or imagined to be, as Libertarians seem to view them) they would have no need of government at all. But they are not.
There are many difficult questions like this in today's world. Though I've argued one side of this issue thus far, I am actually sympathetic to both views. The difficulty we face is how to restrain an increasingly out of control segment of society without unduly restricting the liberty of the remnant of responsible citizens. Unfortunately, I think John Adams nailed it. Once enough of the people turn from God and from reliance on His principles, the Constitution loses its power to govern them.
The original out of control segment of society was the federal government that created so many of the problems that you allude to. The feds must be restrained to work within the Constitution. Many of the problems will go away when that happens.
The Constitution leaves police powers to the states. That should include selling raw milk or not, firearms regulations, drugs, etc...
The government isn't the solution. It's the problem.
/johnny