No doubt the kids’ conduct serves to mitigate his responsibility, but ultimately the driver is wrong on this issue. He willingly accepted the job, and in this case was derelict in his duty.
He willingly accepted the job, and he willingly quit. His timing wasn't the best, but unless under contract, he, like nearly every working American, has the right to quit a job any time he wishes.