There will be significant costs whether you stay or leave. You must choose the lesser of two evils, which is withdrawal if there is no hope of achieving the objectives you originally tried to achieve. No sense becoming bogged down indefinitely lossing lives and national treasure.
It is a poor definition (not every dictionary is good). Quagmire comes from quag + mire. Quag is related to guaky (shaky unstable), mire stands for sinky bog. If you step in a quagmire you are lucky if you loose only your shoes. It is hard to withdraw from it.
From the actual dictionary: "a situation that can easily trap you so that you become involved with problems from which it is difficult to escape" ( Cambridge Dictionary of American English on quagmire)