One would assume this is a simple 4th and 5th Amendment argument, but in your defense, the "right" to travel abroad is not enumerated in the Constitution.
Since 9/11, Congress has passed many laws, such as the Patriot Act, giving broad powers to the US security state, and DHS has implemented many other regulations, such as "Real ID" and "No Fly Lists." Another is a DHS rule that went into effect in February 2008, under which all travelers now require the express prior permission of the U.S. Government to board any aircraft or maritime vessel that will enter or leave the United States. That "permission" is represented by your passport.
The Supreme Court has never ruled against such DHS actions, and Congress has only given more power to DHS to create them in the name of "national security."
So we are in a "no-mans-land" on this issue.
Do not doubt the (neo) marxist DC cabal will use this power ever-more in the future unless actively denied the power.
Protected or not, one can buy or rent their own plane and largely travel anywhere that will allow the plane to land.
Passports were not even required until 1952. The right to travel was not even a question. It was so normal it was not even enumerated.