You don't understand. I am not against divorce nor do I believe there is a complete prohibition against it in the Bible.
However, without a doubt, there is no excuse for your adultery when you have sex with someone else while your vowed spouse is still alive, even if he or she left you in a divorce.
Your spouse cannot make you break your vow to God. But you can do that when you have sex with another.
The vow was already broken.
I think that is the source of the disagreement. If your vowed spouse divorces you, they are not a vowed spouse any more, even if they live another 100 years. Your new spouse is your vowed spouse, and having sex with them is not adultery.