[azhenfud]

Some were used as ballast for sailing ships.

[GSlob]

I believe stones were used for ballast: cheaper (so one does not have to make them), more dense than brick (i.e. take less space), and available if not at the point of origin, then at almost any destination.