The key words there being "at pleasure", words I'm certain you will appreciate since they come from your hero, a man you've defended here at great risk, the Father of our Constitution, President James Madison, right?
You focus is too narrow. Read what Madison wrote again. For the first time.
“The compact can only be dissolved by the consent of the other parties, or by usurpations or abuses of power justly having that effect.”
Beyond that, I can't believe that Madison (though he didn't sign the DOI) didn't believe that just powers are derived from the “consent of the governed.”