I don't think you're correct in that. Article IV, Section 3 says "The Congress shall have Power to dispose of and make all needful Rules and Regulations respecting the Territory or other Property belonging to the United States..." without placing any restrictions on the size or location.
That says Congress in in control of disposing of new territories. It doesn’t say it’s supposed to keep control of them.
And, by the way, Congress has permitted its power to be usurped by the Executive....the Interior Deptmt and the BLM.