IIRC, it was caused at least in part by cleaning up the smut palaces in Times Square.
The main reason was that he refused to continue with the "business as usual" attitude that had led NYC down a toilet for the better part of forty years. He shook things up, he changed things, and there were a lot of people invested in the status quo who found that very frightening.
People ask me why I support Rudy, and I'll tell you, the difference in NYC now as opposed to how it was when I was here before is like night and day. Rudy did that, and it's an accomplishment that I deeply respect.