If you wanted to build that building using concrete made from sand and gravel that you stole, because you thought it would be cheaper to steal it on site than buy it elsewhere and have it shipped in by truck, you would be in violation of the property rights of the true owners of the sand and gravel.
That still doesn’t make any sense. If I dig a hole to make a foundation and a basement where would the removed soil end up? Usually it gets trucked out and dumped somewhere. I’ve never heard of anyone needing mining rights to do that though.