So simple even *I* can understand it.
Actually, most people don’t understand it. According to Jewish law, and therefore the original meaning, adultry is...
“Sexual intercourse of a married woman with any man other than her husband. The crime can be committed only by and with a married woman; for the unlawful intercourse of a married man with an unmarried woman is not technically Adultery in the Jewish law.”
http://www.jewishencyclopedia.com/articles/865-adultery
So even if you don’t accept divorce, if the man marries a woman who previously was never married, there is no adultery going on, according to the original meaning of the commandment.