He knew of it, sure. He was, after all, Minister of Armaments. He didn't make any decisions related to the Holocaust or slave labor; and, the Nuremberg prosecutors could make no connection with Speer. Many from that period will tell you about the compartmentalization of German offices. If it wasn't your business, you didn't open your yap.
Von Braun also knew of the camps and the slave labor. He didn't have any control over it.