Honestly, I took my marriage vow before God seriously.
That's great and that really is one of the first steps to having a successful and affair proof marriage (there are also other important steps that come long before one gets near the alter). However, one's life has to be kept going with that relationship to God. Note that in the verse below in Matthew, it does not say that you will be kept from temptation because you love your husband. It says that you need to 'watch and pray that ye enter not into temptation'. This essentially says that there is no person alive who on their own can handle temptation (if not affairs, then something else) - the devil is just too powerful. If there is any doubt about that, check out how 1 Peter characterizes the devil and his objectives. However, by praying for help and guidance, one obtains the strength of God and that far exceeds the strength of the devil. And contrary to what you said the other night, you can trust God even when you can't trust yourself. This has to be the case because otherwise one believes that they can manage on their own just fine - this is the original sin of pride and it heads for a fall.
Matthew 26:41 Watch and pray, that ye enter not into temptation: the spirit indeed is willing, but the flesh is weak.
1 Peter 5:8 Be sober, be vigilant; because your adversary the devil, as a roaring lion, walketh about, seeking whom he may devour
Philippians 4:13 I can do all things through Christ which strengtheneth me.