And a flag isn't? If I go out to Wal-Mart and buy a flag with my hard-earned paycheck and then burn it on my lawn, seems like it's all private property to me.
You're right; it isn't rocket science: you can't attack people for engaging in speech you don't like. You just can't. End of discussion. It doesn't occur in a civilized society.
After picking yourself up off the ground, dusting yourself off, and locating your teeth, if you want to take your flag and skulk off to an abortion clinic to burn it or go home to desecrate it in private, have at it.