Then we disagree.
You can make a logical argument that the contractor shouldn't be required to get insurance that covers damage to the person who hired him, since that person has the choice of hiring contractors with or without insurance.
However, a person walking by a construction site has no such choice. If a contractor doesn't have insurance and is structured in a way designed to protect its assets from legal judgments (which is very easy to do), the pedestrian's injuries end up being paid by the rest of society. So, the negligent party escapes the consequences of its actions.