The constitution allows for no conflicts. Your argument removes a specific limitation. Not saying our activist courts will not buy it, but it does negate a part.
You're right. Article IV is pretty clear. And if you read the Oregon Statehood Act you would see that Oregon gave up any and all claims to property the federal government retained when they were admitted as a state. It's all pretty cut-and-dried.