However, he left the country to go to the Barbados around 1812 in order to restore his health (after being beaten by a mob in Baltimore while trying to defend an editor friend's right to publish anti-war articles) and possibly to escape creditors as well. He left his wife and young children behind in Alexandria, Virginia, so young Robert didn't have much contact with him.
In 1818 he decided to return to see his family after realizing his health wasn't improving. Along the way he took a turn for the worse and asked the ship's captain to put in off the coast of Georgia. Ironically, he found himself at the home of his former comrade-in-arms, the late General Greene of Revolutionary War fame, where he died.
Ironically, during the Civil War, RE Lee ordered cannon batteries to be built along the Potomac River. One of these batteries was located on his father's birthplace.