There was a time in the 80s & 90s that I would go to the Bronx to watch a Yankee game. Even then, the Bronx was a zoo.
The last time I was in NYC was 2002. Rudy Giuliani had cleaned things up pretty well by that time.
I probably wont go out of my way visit Manhattan, but if I was there on business or was passing through and could grab a few extra days to stay, I would take it in.
Your point about crime is not untrue, but if you look at per capita states, you are more likely to suffer at the hands of violence in St.Louis or Detroit, and there are fewer reasons to visit those cities than NY.