No, I'm sure those same men had no problem having sex with white women, either. But some of them weren't married, and white girls who did that sort of thing outside of marriage were considered tramps and excluded from polite society. And some of those men were just plain old adulterers.
Your point? Nobody's claiming that all men respected all women at all times, at any point in human history.
My point was that only white women's feminine virtue was celebrated and put up on a pedestal. There is nothing desirable about that sort of bigotry.