Joe’s left. Our (viewers’) right.
I think it was just due to where the debate moderators were sitting. When Trump was talking, Joe looked to his (Joe’s) right, where Trump was. But when he was answering a question from the moderator, he seemed to be looking slightly to his left.
Same with Trump, reversed - when Trump was responding to the moderator, he seemed to be looking slightly to his right.