Celibacy was not an absolute rule for Roman Catholic priests until the mid-11th century, and it had something to do with the idea of divided loyalties-it had been proposed in the early 4th century by some clerics from Spain, but was rejected at the time.
Richelieu was also said to have multiple mistresses and to have spread his seed far and wide.
Wolsey, so far as we know, stuck to one wife and fathered a son and a daughter.