Health and sanitation issues?
Foods that are usually banned from cottage industry production are those which are prone to spoilage and contamination unless very strict temperature and handling protocols are observed. Think milk, eggs, seafood, and meat. That is the reason why custards such as pumpkin pie or chocolate cream pie are usually not permitted. If it requires constant refrigeration before sale it is not going to be considered a legitimate cottage food.
The limitation to direct sales to the consumer is because these laws are designed as an exemption to requirements for those who produce food on a wholesale basis for resale. It was meant to allow producers to sell at farmers markets, bake sales, church bazaars and the like. It was not meant to permit mass sales to Minnie’s Emporium.
Fresh produce is probably not permitted because sales of produce come under the purview of the Department of Agriculture and not whatever authority has jurisdiction over prepared food sales. I know in Florida a grower’s permit is required. These are (or at least used to be) free.
St. Mary's County, Maryland, had a case of a long-traditional stuffed ham church dinner being the source of a huge case of salmonella poisoning, due to the way the hams were prepared. This annual dinner had become so well-known that as many as 1,400 people attended it. A recipe for disaster. If I'm not mistaken, the restaurant-style kitchens in large churches can be inspected by the county, but because so much food was needed, several individuals prepared some of the food in their homes and there's where the bacterial infection problem lay.