I believe they should. However I recall a situation when I was a kid that drove neighbors crazy. I lived in an old neighborhood, it was a commuter neighborhood from the turn of the century where wealthy Philadelphia busnessmen lived and commuted into the city. So it was full of old large homes. Then one burned down and a new owner bought the land and build an ultra-modern house that looked like a pyramid. It was a nice home I guess (I thought it was ugly but that's a matter of taste) but it looked totally out of place in the neighborhood it was in. I always wondered why if they liked modern architecture so much, they chose to live in (and change) such a traditional neighborhood.
The way to control things such as that is to have zoning and covenants to not allow such a building that is so radically out of place with everything else.
However, having covenants and zoning stomps all over the property rights of the individual. I oppose covenants and zoning in almost all cases. Having lived in a neighborhood with a practicing "community association" board, I would never do so again.
I'm about as libertarian and pro-property rights as they come, however I do concede that I see some wisdom in zoning laws when it comes to historical neighborhoods.
There is a lot of character and neighborhood cohesiveness that can be completely undermined by one ugly, out of place home built in a historical neighborhood.