Same here. I played during my Air Force careers, because it was more or less expected. There were always numerous tournaments with the brass and community leaders. I still play maybe 3 times a years as a favor to friends who need a 4th. But I never really got into golf.
Many times, it's an excuse to drink. I've golfed with many people who drank before, during, and after the game. Not everyone mind you, but enough.
Also, if you are a hard core player - you are going to be spending a hell of a lot of free time away from your wife, kids, and other friends and family to play the game.
Golf doesn’t seem to have many in between, it’s either a passion or it’s tolerated.