Very few Spanish speakers were in those territories. There were many languages spoken including English. So much so, that it is not correct to try to lay claim that they were Spanish speaking territories.
Mexico laid claim to those territories but it did not have much if any presence there.
You put it better than me.