Until your owner sells your children.
At the risk of diverging even further from the point of this thread, the virtual slavery of Northern factories probably fragmented as many families as the express practice of selling slave children did in the South. Chattel slavery demotes one group from enfranchisement, to be sure. But in doing so, it also places the entire burden of that group's well-being on the owners. And no slave holder with half a brain would risk damaging a piece of property as valuable as a field hand by neglecting him or physically abusing him.
No such constraints compel the factory owner. If a serf is injured on the job, he can be discarded and another put in his place. The injured or dispossessed employee is left to fend for himself.
One of the traditions the South embodied was that of noblesse oblige. In shunning the South's gentility, the North also sacrificed the moral obligation that comes with great power. In the name of Mammon, all things are allowed.