He knew, by the inverse square law, that if he stayed far enough from women, he wouldn't be in danger of being attracted to one.
You are correct, of course.
An alternative explanation: his women were not massive enough to attract him sufficiently.