“The whole concept of musical keys has a lot of math-like elements to it.”
Not so much the key, but the time signature. Considering time signature, music IS math. I knew fractions when I was four, before kindergarten.
The Renaissance composers, especially the big names like Josquin and Busnois and Ockeghem, worked out mathematical puzzles in their music. Story is that Guillaume DuFay's motet "Nuper Rosarum Flores", written for the dedication of the Florence cathedral, with Brunelleschi's great dome, duplicates in its intervals and structure the proportions of the dome itself. Ockeghem wrote a Mass "cuiusvis toni" - "in any key (mode) you like." You could play it in any key and it would work.
The amazing thing was that they could make the music sound so good while working out all these mathematical puzzles.