The visible erosion shown on the daylight videos was soil eroding.
Rock is used when it’s a resource that is available. There are plenty of earthen dams in the SE US that are nothing more than a compacted clay core with silty loam soil as the shell on either side.
.
The dam was built against undisturbed native limestone.
The face had soil overburden, and that is mostly what erodes first, but the limestone can also erode, especially if it is exposed to weather for a long period.
It “chalks.”