Instinct would drive polyandrous men to kill each other off until only one remained.
From what I've read, polyandry often occurs in situations where a bunch of brothers share a wife (fraternal polyandry). If they can't be the dad, at least they can be a uncle. Sort of like a simultaneous levirate.