Whatever area is provided to protesters in which to vent, is a public place, unless they are forced to protest only in their homes or in the depths of some isolated cavern.
Free speech doesn't give us the right to be heard-just the right to freely speak. Not even the President can be forced against his will to listen to us. He has rights too.
Removing peaceful protesters from any public place, while others remain, is a violation of free speech and assembly.
Think of it this way. You have a house. You have a right to be in any part of your house. Someone forces you to stay in your bedroom. Have your property rights been violated? Your theory says no.
Free speech doesn't give us the right to be heard
No one has to listen to the protesters, including the president. Just ignore them. Forcefully removing them however, is entirely another matter.