You are very right. Sometimes, there is a fine line between exercising ones rights, while not stepping on the rights of others while doing so. The neighbors have the right to quiet Sundays at home without all the traffic, car doors slamming and whatnot. jmo
They have no greater right to peaceable Sundays without traffic than others do to peaceable Thursdays without traffic, or peaceable Fridays without traffic, or peaceable Saturdays without traffic, etc.
The LAW applies to all on all days at all times.
When the law singles out RELIGIOUS ACTIVITY, that's simply prohibited by the Constitution!