That was his stepmother. His father he never saw again and did not attend his funeral.
Stepmothers aren't family?
Lincoln's brother died when he was three.
His mother died when he was nine.
His sister died when he was 19.
The only family he really had after he left home was his father and his stepmother. He was not close with his father, but he did see him again. And he kept in touch with his stepmother and visited her often when she was widowed and lonely.
His father he never saw again
He visited his father in 1849.
But he didn't tell people that his father wasn't his real father. And his father was not retarded.
His father was not an educated man, but he was a successful farmer and a trustee of his Baptist congregation.
David Herbert Donald relates that Lincoln visited his father when he was ill in 1849, two years before his death.
Now, something folks usually ignore ~ Abe Lincoln's parents had been members of what is usually called "the Primitive Baptist Church". In the old days it was pretty much the same thing as the Christian church (Disciples of Christ) and not like the Christian church (Independent) that grew up out of the Stone movement although some writers try to link it to Stone as well.
Basically most of these congregations were Baptists who met on Saturday but adhered to the order of worship for the Presbyterians as modified by Alexander Campbell and his ministers.
Lots of their graveyards have no headstones. There are no lists or maps telling you where people are. You have one of these guys in your lineage you are out of luck using normal research methods. Abe most likely knew there'd be no gravesite he could visit. No doubt the gravesite has been marked in later years by someone interested in doing that sort of thing.
Unmarked graves were a common American practice by the "cutting edge" folks back in the firt half of the 19th century. This practice lingered in places. Some of my cousins who live up in Alaska get upset when they come down to do pilgrimmages to the ancestors' graves in Brown County and discover there are no headstones. Last I heard they're raising a fund to mark those graves.
Whatever you read about what Abe Lincoln might have thought about his father must be taken with an enormous grain of salt. He didn't tell anyone, and most the stuff in the standard histories is sheer supposition by alleged historians.