Good point. Also, if Jesus had a wife and family they would have been revered and perhaps worshiped as deities themselves. The Catholics already do that with his mother; can you imagine if he had a wife and kids, grandkids, great grandkids...?
An urban legend perpetuated by the ignorant.