If Obama was born August 4, 1961, his eleventh birthday would be in 1972.
I was thinking you had a really good point here concerning Obama's lack of memory for details, but on rereading your post I wonder if the error does not arise instead from Obama's poor sentence structure. Perhaps he meant to say that his father's visit was before his eleventh birthday, rather than the later trip.
I vote for poor sentence structure ... he was trying to write his bio like a novel.
“It was during the summer after my father’s visit to Hawaii, before my eleventh birthday”
... His eleventh birthday was August 4, 1972.
The timeline fits if his father visited him Christmas 1971, when he was 10 and 1/2 years old.
Poor sentence structure doesn't explain how they watched the Watergate hearings every night on TV in 1972.
The legend is his father visited Hawaii when he was 10 (Christmas 1971). It doesn't make sense to refer to his father's visit as occuring "before my eleventh birthday". Either way, "the summer after my father's visit to Hawaii," would still be 1972 and "before my eleventh birthday" would also be 1972.