And, if the guardian commits crimes, he can be disqualified. He committed the crime of adultery.
From the Florida State Law on Co-habitation
798.01. Living in open adultery
Whoever lives in an open state of adultery shall be guilty of a misdemeanor of the second degree, punishable as provided in s. 775.082 or s. 775.083. Where either of the parties living in an open state of adultery is married, both parties so living shall be deemed to be guilty of the offense provided for in this section.
Bingo!
Spouses are not the guardians - they are the proxy.