Supply-and-demand economics would explain this pretty well. I don't know what sum would usually be demanded as a bride price, but I would expect that even if polygyny would be permitted for those who could afford it, it would in practice probably not have been terribly common.
BTW, the Old-Testament rules surrounding adultery were not symmetrical between men and women: adultery was specifically the crime of a man (married or not) having relations with another man's wife. A man who had relations with a virgin could be compelled by the girl's father to marry her, and such compunction would not be affected by any existing marriage the man might have. There are clearly logical reasons behind the assymetry in Old-Testament law; some of those reasons still exist while others do not.
But, it is clearly not condemned in the Bible. Anyone who makes that argument can easily be repudicated.
And Lawrence v Texas makes it clear that any law prohibiting polygamy (in at least the informal sense) is unconstitutional.
There are plenty of good arguments against polygamy.
It's just that the Bible isn't gonnna be much help..