It actually is harmless. But some people do not stop at a few. Petoskey stones used to be a cool thing you would hunt with your kids and bring one home. And then the professional hunters showed up and mine the beaches for them to sell on-line.
Doing things as an individual rarely does harm. It is when it becomes commercialize that stuff gets depleted.
Glass Beach in Mendocino County (CAL) is a prime example with 'artists' hauling off the ocean polished glass by the bucket load.