I agree that cats aren't as dangerous as some dogs are, but they can spread diseases and they can destroy property, and they can be a nuisance. If your cat happens to use your neighbors' flowerbed for a litterbox, that could be dangerous if your neighbor was pregnant.
Not to mention you are increasing the cat's chances of picking up all kinds of nasty diseases like panleukopemia, feline herpes virus, and feline leukemia, and of course getting mauled by other animals wild or not, getting hit by cars, and getting tortured to death by people. All of these things happen on a daily basis. It's so bad now that even out in the country your cat isn't safe. Cats living indoors live much longer than cats living outdoors, and the above are only a few reasons why that is.