Jews (Safardi) took a big hit prior to 1492 when Jews who would not convert were expelled from Spain, mostly to north Africa. Then there was the great plague which decimated Europe between 600 and 800 years ago. If Jews were condemned to ghettos and very poor villages they may have died in greater quantities than the Christians because of more rats and poorer nutrition. How did Jews feel about cats as pets?
Jews were blamed for the plague due to their apparent lack of it, so that probably isn’t it.
Don’t forget the destruction of Jerusalem by the Romans. The only tribe left would have been Judah and parts of Manasseh. If they died or were enslaved in the numbers estimated then there would have been very few pure bloodlines left.