If “to golf” is s a verb, “golfing” would be its present participle, so one can go golfing, just as one can go fishing, bowling, skydiving, etc.
I can see that you are really not much of a golfer or you WOULD say "Let's play golf" instead of "Let's go golfing."
Just listen to the announcers at a golf tournament. They NEVER say "golfing" no matter WHAT the grammar books say.