In most cases the native American’s let the settlers into the country. sometimes they killed them. Owning territory was recognized throughout the world, including native Americans, as a right of conquests. We conquered the land and we hold the land. Therefore it is ours. If someone else wants it then they have to conquer it from us.
Or, we can just give it away, as we seemed to be doing until Trump came along, and maybe still are.