You make a good point. But I don't think it is that hairy at the undergraduate level. Usually the last math course most people take as undergrads would be differential equations, which sort of tops out the applied calculus sequence. Unfortunately, that is just a hair short of the "cool" stuff.
It is a shame that the calculus sequence is not followed by a semester or two in "abstract algebra" (or "modern algebra" if you prefer). That should be sufficient background to follow these developments in physics, the same way an ordinary chess fan can follow and appreciate the game of a grand master.
For the contemporary world such a course is really invaluable and touches not only on the theoretical but also solidly practical applications like cryptography as well. And a lot of folks find it easier and a breath of fresh air after slogging through the calculus long march.
Thats me, and most engineers probably, topped out at differential equations, the calculus long march indeed.