I recall from international law that the reason foreign travel is a privilege is NOT because you can’t leave the country you are in. It’s because no other country is obligated to let you come in.
There are large parts of Antarctica you can go to without a passport.