I have worked for a Catholic Priest for over three years now, and the answer is no. They get days off just like everyone else, and they don’t always have to wear their clerical clothes then. Sometimes it is for their own protection. There are times when they travel that it is also necessary to take identification (picture) without the collar, again for their own protection.
Of course! Like you, I work at the rectory, 2 days a week, and when needed for special meetings. When our pastor's family arrived from Lebanon last month, he drove down to JFK to retrieve his family, dressed in secular garb. When I arrived this morning to work on the bulletin, he was again dressed in secular clothes but showered, shaved and changed into clericals before driving to a local catholic hospital where he says mass. On those occasions when the bishop comes, he will usually wear a cassock over his clericals.