He did a lot more than "speak up". The convents and monasteries around Rome were full of Jews. The Pope's own summer house at Castel Gandolfo was the place of refuge of a number of Jews; some Jewish babies were even born there. When the Nazis demanded a ransom of 50 kilograms of gold for some Jewish prisoners, the Pope provided 15 kilograms from the Vatican reserves. There are many other examples.
When John XXIII, who was then the Papal nuncio in Istanbul, was credited with saving thousands of Jews by giving them false baptismal papers, he replied, "You should thank the Pope. Everything that I did, I did on his instructions."
A piece of history not very well known....
There was a full fledged assault on the Jewish community near the Vatican. Read the book Growing up Jewish in Fascist Italy. The writer is not fond of Pius XII.