hermits?
There were several people in the early days od christainity when many would wander into the desert to live in absolute solitude. As it turns out, they end up getting a bunch of followers who just won't leave them alone and a monastery is set up. One such prominent person did leave his wife and kids(with or without their permission I don't know). Anyways, I'll try to find the guy and get the bio(saw it on a doc once, it's a little sketchy.)