Something about the place always scared me. We love Charleston and Savannah.
I always thought visiting N.O. was the modern equivalent of visiting Manhattan's notorious Five Points (of Gangs of NY fame) area for amusement, as many of the better classes did in the 1800's.
I do admit that N.O. did have some nice old architecture, even if it was defaced by the addition of all those wrought iron railings later on.