I don’t buy it. I see this used as a weapon for the most vile people to beat their spouses into chattel. Diversion from worshiping God is often termed adultery. I think the definition is broader that sexual infidelity.
Think what you want, but in this context, the text is addressing physical treatment, not emotional or spiritual fidelity. There is no figurative sense to this. A Hindu woman turning from idols to God has no excuse for leaving her Hindu husband, to whom she still owes allegiance (for instance).