That 128 must be a very inbred group.
Every person has two ancestors; and 600-800 years ago is roughly 30 generations. Those 128 should have had 2^30, or 1,073,741,824, discrete ancestors living at that time; 350 is a branchless family tree. *<];-’)
First, keep in mind that Ashkenazis and Sefardis shared ancestors. The Ashkenaz had a bottleneck, but had plenty of Jewish relatives in other branches of the family.
Also, I have heard from two traditional Jewish teachers, from different learning traditions, that, over the centuries, millions of Jews were killed in the Christian world before the advent of Nazis. This would seem at least somewhat consistent with some estimates of the percent of Jews in the pre-Christian Roman Empire.