Smartphones need all that crunching power for the GUI and for multithreading numerous tasks simultaneously while running relatively inefficient code compiled or interpreted from human-friendly computer language. The pieces of critical code which actually interact with the hardware are mostly 'drivers', but in the spectrum of modern software drivers represent a relatively small patch. Most code is designed to interact with us rather our hardware.
In the days of Apollo virtually all the software interacted directly with the hardware, it was all drivers so to speak, and the very tight code was painstaking created manually by humans working at the machine language level. Tight code is almost magic.
I first learned to write programs in the late 1970s in FORTRAN using punch cards. I decided that computer programming was not for me because even then software was still, for the most part, painstaking created manually by humans working at the machine language level.
I have nothing but admiration for the programmers in the 1960s who managed to write all of the software necessary to send men to the moon in back on computers that used reel to reel tape drives for storage and that had less total computing capacity than a modern pocket calculator or smart phone.