No, you can leave him. But the damnation part comes if you divorce him and choose to live an adulterous lifestyle by marrying another. His words, not mine.
Thank you for the clarification.