I guess how would someone know you have cats if they are indoor cats? Do they see a cat in the window? See the homeowner carry in litter?
The issue is - what is the injury to the community with having cats that are exclusively indoors?
The question remains, what would be too many?