I'm a native New Yorker, white, born in '48 and, no, it was not relatively safe back then. I remember reading about white people, usually tourists, getting lost on the subway, straying into Harlem and getting raped, robbed, severely beaten, killed. It happened all the time but it wasn't as bad as it's been in more recent years. Democrats have made it much worse with their race-card politics. But the racial hatreds have always been there.
I wandered into Harlem in my car while visiting NY in the early '70's. Luckily, there was almost nobody on the street, but I couldn't get out of there fast enough. Scariest place I've been.