It's rooted in the ashes of penitence - which are found all over the Old Testament.
I watched the Pope offering the Ash Wednesday Mass on EWTN. He and the co-celebrants sprinkled/crossed the top of the heads of those who came forward. That was interesting. Further, a Cardinal came up, removed the Pope's cap and did the same to him.
The congregation did not line up either. Instead, they sort of milled around at the front and the Priests reached out into the crowd while the Pope sat and did the same to a long line of Priests, Friars, and the like.
The first mention of ashes are in relation to the ashes Moses tossed into the air, which then became boils on the Egyptians.
In another instance, Tamar put ashes on her head after she was raped by her brother.