So only Catholic Seminarians were forced to join?
Yes, it was mandated for Catholic youths in Semanaries per the Reichskonkordat, the concordat between the Holy See and Germany, guaranteeing the rights of the Catholic Church in Germany. It was signed on July 20, 1933 by Secretary of State Eugenio Pacelli and Vice Chancellor Franz von Papen on behalf of Pope Pius XI and President Paul von Hindenburg respectively. Membership in the Hitler Youth was a requirement to counter or "balance" the influence of the Catholic Church.
Beyond that, beginning March 25, 1939, all German males 14 years old and older were required to participate in the Hitler Youth as a condition of enrollment and receipt of tuition in all schools. Following his 14th birthday in 1941, Ratzinger was enrolled in the Hitler Youth, as membership was legally required.