“Should”? That is a legal question.
There have been male-only schools that have seen women attend. And women who’ve been outraged when men are permitted. And historically black only universities that didn’t want outsiders.
They are still segregated, whether legal or not.
it’s a negative connotation and the implication is that there should be pressure to desegregate the ‘segregated’ boys schools or the female only restrooms or the men’s track and field team.