I thought it was pretty straightforward. A high density conclave in a small town just doesn’t belong there and I don’t care WHO you are.
Now, I would change my mind in a heartbeat if there were other high density areas in this small town and no one minded. But I don’t see that here. They are literally talking about building a project for Jews.
Well, God cares.
There is also probably a religious reason for this. They want to be able to visit one another on a Sabbath (Saturday) without undertaking what they view as unseemly travel. And they can also carry books, food, etc. around with them that they wouldn’t carry on the street. If they are observing it out of conscience towards God, then God honors that.
If opposed enough they will go away, and some other locale will get them. And the potential blessing will move on as well. I don’t get the basis of “just doesn’t belong.” Who is the king of belonging here?