Because the drivers should not live in Kernel land (use Kernel Memory Space) short of s Bus going drivers should not touch the core of the system. If a video driver fails I should lose my monitor and get a ton of crap put out to an error log.
But the whole reason you have a driver is the O/S by itself doesn't know how to interact fully with a piece of hardware.
What they understand is that a Turing machine teaching you the essentials of an OS. Other than the system Bus, Memory, Processor, and maybe some storage you should not *need* anything else to keep a system running. If the monitor, mouse, keyboard, speakers, Ethernet, .... go the system itself should not crash.
Something along the lines of... well, we can add another layer of abstraction and keep the system more stable, or we can not and have faster video. Seeing as there are a lot of people who rely on NT to deliver good performance on applications such as Microstation, they chose the latter.
NT was there shot at a server environment, people run web servers, database servers, mail servers and the lot on them. It was a long time from designing NT to releasing an NT Kernel intended for the desktop. So when putting together a server whats more important fast flashy graphics or stability (Hint on my *nix servers I rarely even install X).
Maybe I'm naive, but I have a feeling this was a hotly debated topic inside Microsoft. Maybe it was a wrong decision, but if you want to debate it you can't ignore the impact it would have on performance.
Sure we can, with Video cards themselves having so much memory all MS needed to do with another layer is provide a solid API (I know well documented API's are hard to get out of Redmond) so they can let the card do all the work.
SGI puts video performance in its own layer and I think they are kinda known for something to do with graphics..