You are mistaken in that. George Washington Parke Custis died in October 1857. Lee manumitted the last of the slaves in December 1862.
From what I’ve read, the manumission process was involved and time-consuming, which was the reason for the delay between when Lee inherited the slaves and when the last of them was actually freed.