Matthew 19:9 says exactly that:
And I say to you, whoever divorces his wife, except for sexual immorality,[d] and marries another, commits adultery; and whoever marries her who is divorced commits adultery.”
As I understand Mt 19 and 5, a man doesn’t commit adultery if he divorces an unfaithful wife. The relevant verses are below.
1. Infidelity is an exception. (Mt 19, Mk 5)
2. The infidel is the CAUSE of any adultery by both the former spouse and anyone who marries them.
3. Adultery is against the former spouse (Mt 10)
4. A woman, specifically, is bound to her husband by marriage. (Ro 7, 1 Co 7)
5. Couples in which both parties are Christians are to remain married.
6. Split faith couples must remain together if the unbeliever chooses to stay together. (1 Co 7)
7. If the unbeliever deserts the marriage, then the bond mentioned in Ro 7 and 1 Co 7 is no longer in existence.
Summation:
1. Infidelity can lawfully break a marriage.
2. “Remarriage adultery” after infidelity is the fault of the infidel.
3. Desertion breaks the bond of marriage as if the marriage never existed.
Mt 19:9 9 I tell you that anyone who divorces his wife, except for marital unfaithfulness, and marries another woman commits adultery.”
Mt 5:32 32 But I tell you that anyone who divorces his wife, except for marital unfaithfulness, causes her to become an adulteress, and anyone who marries the divorced woman commits adultery.
Luke 16: 18 “Anyone who divorces his wife and marries another woman commits adultery, and the man who marries a divorced woman commits adultery.
Mark 10: 11 He answered, “Anyone who divorces his wife and marries another woman commits adultery against her. 12 And if she divorces her husband and marries another man, she commits adultery.”
Ro 7: 2 For example, by law a married woman is bound to her husband as long as he is alive, but if her husband dies, she is released from the law of marriage. 3 So then, if she marries another man while her husband is still alive, she is called an adulteress. But if her husband dies, she is released from that law and is not an adulteress, even though she marries another man.
1 Co 7: 10 To the married I give this command (not I, but the Lord): A wife must not separate from her husband. 11 But if she does, she must remain unmarried or else be reconciled to her husband. And a husband must not divorce his wife. 12 To the rest I say this (I, not the Lord): If any brother has a wife who is not a believer and she is willing to live with him, he must not divorce her. 13 And if a woman has a husband who is not a believer and he is willing to live with her, she must not divorce him. 14 For the unbelieving husband has been sanctified through his wife, and the unbelieving wife has been sanctified through her believing husband. Otherwise your children would be unclean, but as it is, they are holy. 15 But if the unbeliever leaves, let it be so. The brother or the sister is not bound in such circumstances; God has called us to live in peace. 16 How do you know, wife, whether you will save your husband? Or, how do you know, husband, whether you will save your wife?
1 Co 7:39 39 A woman is bound to her husband as long as he lives. But if her husband dies, she is free to marry anyone she wishes, but he must belong to the Lord.
2 Co 6: 14 Do not be yoked together with unbelievers. For what do righteousness and wickedness have in common? Or what fellowship can light have with darkness?