I suspect most men feel that way intuitively--but have been brainwashed out of it, as people have an equally strong inborn desire to feel they belong to a group--a society; our modern society tells men and women that following certain of their inborn inclinations is bad, so people repress those inborn feelings to fit-in.
This creates one of many innter-conflicts that account for the mental problems of moderns.
We have perhaps gone from the too prudish to the opposite extreme in the past 150 or so years.
In my opinion.