Forgiveness doesn't mean staying with a person who is not keeping his/her marital vows! You can forgive another person but that does not mean you have to subject yourself to his/her behavior, esp. if it's harmful and immoral.
I've forgiven my FIL for his behavior to our family (NONE of it was sexually inappropriate, but he's caused LOTS of pain), but if I have the choice I will never speak to him again, never eat another meal in his house, and never let him enter my home again. I've forgiven him, but I will NOT subject myself to him.
Do as you will.