If you really focus on the video, you can see Carter hesitate before uttering “boy”. I see that he was pondering whether to call him a man or a boy.
He chose “boy”. Man would have been the correct choice.
I think he was hesitating because “boy” really was the correct choice, but he was worried about saying it. He was talking about Obama being raised by a single mom and his grandparents.
Carter is loathsome, but fair is fair.