The founding fathers felt differently. Robert E. Lee swore the same oath, but felt that his first loyalty was to his home. That is simply a basic rule. You don't turn against your kith and kin....or maybe they DO up north!
Why would I have sworn allegiance to a State that supported slavery & told me to go to war for that cause or leave?
Why should I swear allegiance to our modern day States that violate our Constitution every day in nearly every way?
-- I chose to honor a defend our Constitution, against ALL its foreign & its domestic enemies, federal, state or local.
And believe me, there are far more State & local enemies of the Constitution than Federal.