I do think the work pool is different. Working outside in the dust, high humidity, daily temperatures in 90's and using a port-a-potty doesn't appeal to workers who have always worked in air conditioning.
Been fun talking to you but it's past time to get up and do something productive. Adios
You are still drawing from the same unskilled labor pool - no different than work hands in horticulture as well or grunts first starting off on construction sites. The individuals in that labor pool have different preferences and priorities (eg: wages, time of day they work, what days they work, FT vs PT, permanent vs seasonal, benefits, physical labor or not, inside or not). The entire group isn't monolithic.
The largest reason you'll have trouble competing with retail/hospitality/restaurants isn't inside/outside work - at least with men - its going to be permanent FT jobs vs seasonal FT/PT jobs. Turnover rates among PT workers is more than 2x what it is for FT workers and seasonal is even higher than that. The second largest is going to be they are also competing far more with illegal labor than in the other industries today.