You could go into black areas in the day time; i.e. to the museum. Blacks had their own schools. Restrictive covenants were allowed which helped make neighborhoods safe.
In short, whites governed and it was generally peaceful. We never really thought much about blacks. My mother was horrified when I saw my first black.(I was about five.) I looked at the lady and said "Mommy that lady is dirty" Mom remembered that all her life. She had to apologize to the lady.
You must have grown up in a different part of So Cal than I did.