Me too. It was safe. I actually went downtown at age 11 on Friday and Saturday nights to volunteer at the symphony in exchange for a free seat. I rode the bus home at 10 pm afterward and never was afraid.
I grew up in downtown Jersey City, not too far from the Holland Tunnel. Just about the same age I would regularly take the Path Train I (We called it the Tubes) into Manhattan and just walk around. No one ever bothered me.
Today, kids of our age reference have to be kept on a leash to protect them from the insidious individuals our loose society has created.