Don't know where you live, but we just put a huge addition on our home.
Our concrete guys were black; the carpenters were white; the plasterers and drywall guys were white; the plumbers were white; the electricians were white; the roofers were the only hispanics in the bunch (and there's was definitely the most grueling job since they work on the roof in the Florida sun).
I'm sure demographics plays a part in this, but to say hispanics are the ones building all the houses, sorry, not in our neighborhood.
You are right, of course, the areas without a heavy Illegal concentration would of course need to use the legal workforce.
But that's the point, many other places have jobs taken from needy people by businesses of all sorts using these cheaper workers.
Eventually this negatively impacts all of us. All those workers earning less money means less money available for them to spend on something your job may produce, then your labor is worth less to your employer so you earn less money.