It was not the adultery fornication which was chronic that was the basis for your annulment.
It almost certainly was was a separate problem which existed before your marriage that resulted in such behaviors, usually designated as “the ONABILITY to fulfill the ESSENTIAL OBLIGATIONS OF MARRIAGE.
I think what’s poisoned the well is this notion that you marry your “soulmate” and that marriage should be effortless.