In an extension of your thought process, why shouldn't I be able to restrict your right to express your opinions because they might raise my blood pressure? Eroding private property rights further in the name of "health concerns" is no different than stifling free speech.
First of all, eroding private property rights further in the name of "health concerns" is different from "stifling free speech." It may be just as bad. It may be just as dangerous to the body politic. But it is different if there is anything to the word distinction. And that's what a community does -- it distinguishes one conduct from another, one transgression from another. That's why robbery is punished differently from murder. The acts are different, and the community distinguishes between the acts.
But I think your proposition is an excellent test of our nation's politics. How do we conduct the test? Professors will decide? Lawyers will decide? A unanimous vote of all three "minarchists" will decide?
I suggest that since it is a political question and we have a chosen political system that we test this proposition as we do others. Why not take it to your Representative or your Senator (or both) and ask them to introduce legislation that will make it equally illegal to "erode private property rights further in the name of health concerns" and "to stifle free speech." While that is percolating through the system, you could also find a case in which you thought private property rights were being further eroded and take it through the state and Federal court systems asking whether or not that was like stifling free speech.
I realize that may seem clumsy, and if you have different ideas about how to test this proposition, I'd like to hear them.