I had the same experience. Went there every year for a conference from about 1980 to 1998 and loved the city. Went back in 2017 and saw everything you said one block from the Wharf, plus drunks sleeping in doorways. I will never go back.
Messed up city