"But if the unbeliever departs, let him depart; a brother or a sister is not under bondage in such cases. But God has called us to peace." (1 Cor. 7:15)
If the wife is judged an unbeliever by the church because of her action, then the husband is free.
However, if the church judges that he was not entirely innocent in the situation, then it is appropriate for him to be removed from the ministry.
Looks like we are in agreement.