Do you remember Lt. Speer in Band of Brothers? He became the commander of Spandau prison where Albert Speer was incarcerated. Major Speer developed a respect for Albert Speer through conversation.
But later there were pictures of Speer at the death camps, which showed he indeed knew of the Holocaust.