That's a very smart observation. Thanks for pointing that out.
On further reflection, this reasoning eliminates the concept of “adultery” entirely. If a participant in a relationship with sex-partner A takes up with sex-partner B, then the relationship with sex-partner A wasn’t a marriage, no matter what the documents say (or A might have believed), so the adventure with B isn’t adultery.
This doesn’t sound, to me, like what God intended, although one can see its utility for a person who wants to have multiple sex-partners without feeling guilty.