Even if ten times as many pieces had been lost, burnt by Saracens etc - that takes the tally to 1/5 of a cross.
That's ... a pretty small forest. And that's all that the historical record actually tells us. Anything else is unfounded conjecture.
It’s a big mistake to underestimate the extent of human folly. I’d easily believe several whole crosses were sold and thousands of fragments sold, resold, donated to kings to curry favor etc. A fool who parted with his money stupidly may try to discreetly dispose of the evidence.