Hmmm, because of my eariler wilder life I can say with the confidence of real-world experince to your "there always is more there", that: "No, there isn't. Lust and back-stabbing infidelity are very simple things. Just that."
Some folks who've had hard times in their marriage, like myself, would say the alternative is keeping a promise you made in front of God, your family and your community. Marriage is tough work sometimes, but there is no excuse for adultery, only reasons it's slightly less despicable than committing adultery on a hypothetical perfect spouse. I don't mean to diminish the importance of affection and sex, but your post reminds me of these lines from The Last Boy Scout...
[Joe has just found out that Mike was sleeping with his wife]
Mike Mathews: Look Joe, it just happened.
Joe Hallenbeck: Sure, sure, it just happened. Could happen to anybody. It was an accident, right? You tripped, fell on the floor and accidently stuck your d*** into my wife. "Oops, I'm sorry, Mrs. H, I guess this just isn't my week".
And let me tell you this: I hate divorce, but the guy who divorces his sucky wife and marries somebody else (even her best friend) without ever having cheated on her is a freakin' hero compared to a guy who cheats, and that goes for guys married to downright hags just as much as it goes for guys married to someone great.