Everybody completely disregards the Byzantines - the eastern half of the Roman empire that kept on truckin' for 1,000 years after the western Roman empire collapsed. That's where the muslims got their hands on the Greek philosophers and mathematicians and that's where western Europe was re-introduced to those works during the Crusades.
I don't think the Crusaders took too many books home. Most of the Greek to Muslim to Christendom knowledge transfer took place in Spain. When the Muslim rulers left Spain they left behind vast libraries - and swarms of Jewish scholars who could translate.