All my ancestors, that I know of, were settlers and colonists, not immigrants.
But that’s not really the point. The point is that, unlike the “Native Americans” we’ve got a choice who we let in.
In most cases the native American’s let the settlers into the country. sometimes they killed them. Owning territory was recognized throughout the world, including native Americans, as a right of conquests. We conquered the land and we hold the land. Therefore it is ours. If someone else wants it then they have to conquer it from us.