There really isn’t that much milk fat in regular milk. And children need it to adequately digest fat-soluble nutrients in the food they eat.
I like your reference to Southern soul food. Someone once told me that southern cooking was mostly fat, sugar, and booze. I know it isn’t true today, but some of the older recipes I have from a half-century ago certainly read that way!
Southern soul food is rural poverty food, which is to say it features staples like corn meal, flour, rice and beans, cheap cuts of meat, and basic garden type vegetables. Also it uses animal fats like lard and butter. It’s actually very healthy if you tone down the carbs (biscuits, cornbread) and play up the meats and veggies.