You are more right than you think.
At one point in the middle ages its estimated that the population of the Iberian penninsula was 10% Jewish. Most were converted to Chistianity after the Almohad invasion and the Christian counterattack that took Andalusia.
From the twelfth to the fifteenth centuries most Iberian Jews converted. The grandparents of Torquemada, the great inquisitor, were Jewish converts. This was typical of the Spanish bourgeois of the time.
And the final decree expelling the remaining Jews permitted them to stay if they converted, and 2/3 to 3/4ths did convert, only a minority even of this minority were expelled.
So between one or another wave of conversions, and subsequent integration into both the populace and the gentry, there is consequently a very great deal of Jewish blood in the Spanish people. Anyone with Spanish ancestry can plausibly claim Jewish descent. My grandmother for instance, was of a family long considered conversos. This claim is a common conceit, fashionable since the 19th century. There are even classic zarzuelas (operettas) on the theme.