Besides, they always have the equalizers. They always have to wait for an 8 am opening of a library or zoo when they leave at 11 the night before. Having a time advantage isn't usually much of an advantage.
That's irritating, isn't it? A team works up a three hour lead, then they have to sleep outside of the next challenge area until it opens. But I guess it's better that way--the show would be boring if one team had an insurmountable lead.