I remember reading an interview where he said the song was about his mother. Yoko was a mother figure to him, too.
About his mother, yes, but he was telling his mother about Yoko in the song.