The vow of celibacy applies only to Catholic clergy. There is a prohibition against fornication (all sexual contact between unmarried persons) applicable to all Catholics regardless of vocation.
And, in case Mikey wonders, the vow of marriage includes a vow of chastity, that is of sexual faithfulness and not to abuse (be unchaste) within one’s marital sexual relations. The monk or nun vows poverty, chastity and obedience. His or her vow of chastity means a vow to be totally sexually abstinent and is predicated on a commitment not to marry (celibacy). Secular priests do not take the poverty vow but their chastity vow presumes celibacy.
Everyone is obligated to chastity. It just means something different if one is married rather than unmarried. If you look at the root word, castitas in Latin, it simply means pure, unpolluted.
If you are going to say that chaste applies only to the sexually abstinent, that chastity = abstinence, then you are saying that marital sex is polluting and cannot be pure.
But if castitas simply means purity, then it applies to abstinent purity if unmarried and to pure sexual relations if one is married, rather than impure sexual relations (selfish, manipulative, abusive sexual relations).