I’ve done both, (Scouts, not SP) and I have to agree with you!! SM is much more admin oriented.
At the mere hint that Stake President might be in my future (or even Bishop), I’ll be joining colorcountry! (Just kidding...sort of.)