Just not using it as your browser will not protect you from its security flaws. It's used by many other applications and Windows itself which give you exposure to varying degrees (heavy exposure if you use Outlook).
Let me get this straight - you are claiming that if you use say FireFox - you are still vulnerable to IE hacks while you are browsing with FireFox.