Says you. I don’t have any such problems like you describe maintaining female friendships. Since I’m into extreme sports and my isn’t, I’ve traveled all over the country with a woman who does share my interests. The closest thing to a flirtation that has occurred is that she claims if something does happen to my marriage, I’m taken.
So...there is an understanding if “something” happens to your marriage you are “taken”. This supports my theory far more that it supports yours.