I'm pretty sure one could do that in DOS.
Essentially an OS could be written into an app to take over the machine.
You are correct; in fact you could write an OS that was a valid DOS application.
I was using TP7 to do this for my OS which I got to the point of being able to recognize commands and change the screen resolution before I shelved the project [due to school and getting stumped on handling memory management*].
* I was looking for a way to make the memory-manager 'tyrannical', and generic [able to handle the small stuff (like variables for the compiler)], in order to cut down on memory-leaks.