If you paid the gardener for the time, yeah.
That's why you're supposed to supervise contractors.
In the hypothetical case presented, you would not be responsible. Example: a contractor you hired to put a roof on your house, leaves the property, breaks into your neighbor's home and steals jewelry. You would not be responsible for his crime or for reimbursing the neighbor's loss. IF, however, the contractor was acting as your agent (i.e. a cable installer who steals jewelry from the house while installing a customer's cable TV lines) and THEN committed the act, you could be responsible.