Because if you lived in a neighborhood you would understand the need to be considerate of the “rights” of everyone in a neighborhood.
Sure, there is a “right” to free speech... but does that mean that your next door neighbor can get on a megaphone at 2am each night and sing Britney Spears songs all night?
Sure, there is a “right” to assemble... but does that mean that your next door neighbor can have a party for 500 of his closest friends each night till 5am?
All “rights” have restrictions placed on them to protect the rights of others.
We are speaking specifically about the right to hold religeous meetings in your own private residence. there should be no restrictions on that. If you think there should then you are perhaps on the wrong forum.