I agree. Let's look at this practically. Why would a nobleman do something that guarantees his serfs being perpetually pissed off at him, when it would be simpler and easier to simply offer servant jobs in the castle to the more attractive young girls, and send the girls back to the peasant village when they get older?
In the real world, then probably even more than now, it isn’t necessary for the baron to rape young women. They throw themselves at him, as a rule.
A young woman who managed to produce a child for the lord could have the child recognized and be raised as a member of the aristocracy, albeit an illegitimate one. Though this usually applied when the woman was a member of the nobility or squirearchy herself. Serfs were pretty much below being noticed.
Although it should be noted that William the Conqueror was a bastard, the son of a tanner’s daughter. That’s pretty impressive social mobility for a single generation.