Probably recycled, as in the case with a lot of the material from much more recent Imperial Roman structures. The longer a pile a rubble lays around, the longer someone has to think of a use for it.
Just rub a rock of equal hardness against your target long enough and it happens.
All it takes is enough people and enough time. No doubt the early Britons had neither so they didn't polish them up.
This stone rubbing trick had been around since Homo Habilis too!