I used to toggle in programs in binary and after a while it becomes like playing music from memory on a keyboard. Not as difficult as you think.
Yes, it's very much like playing music! (I play piano).
My first hands-on experience with a small computer was a PDP-8e at college, with a paper tape punch and a teletype. 4K 12-bits words of core, and front-panel bat-handle toggle switches for programming it from scratch.
The reason I mentioned the story about Cray is that I toggled the PDP-8 boot loader in by hand (in octal) many times -- it was just enough to run the punched-paper tape reader so you could read in the binary image loader, which in turn could load in.... the line editor. Or the assembler... or your program...
The PDP-8 boot loader was probably tiny compared to the one for the bigger machines, but it gave me an appreciation...