The Constitution says “their respective numbers”.
Invaders aren’t theirs and must not be counted.
The Constitution, as amended, says everyone must be counted.
It then also specifies that apportionment should be reduced by proportions of certain categories of non-qualifying people who are there.
For some reason, most of those arguing the first point, tend to leave out the second part of the same amendment.