Marriage isn't slavery, it's a social requirement for the perpetuation of societies. It is at it's foundation much more beneficial to women and children than to men. That's by design. For societies it doesn't matter who is the father of the children, only that children have fathers (I don't mean sperm donors). Modern America has grafted some more inequities onto traditional marriage and attempted to define it out of existence. In the past, men could reject taking responsibility for bastards. I believe that should still be the case along with custody and the unequal balance of power in a marriage.
Case by case, in a society where marriage is strong there are going to be some hard cases. This particular one is one of them but it's a bad idea to alter a foundational construct because of hard cases.