Comets moving through the asteroid fields have swept many small asteroids with them. It should be no surprise that their number is diminishing.
The mass of comets heading through the asteroid belt over the ages could exceed the mass of the asteroid belt, but the two numbers together wouldn’t equal the mass of our Moon.