There were undoubtedly a great many Israelites of other tribes who immigrated to the southern kingdom over the centuries they existed together. Especially during the most severe times of apostasy in the North, and when the Northern Kingdom was in the process of collapsing. Members of losing political factions in the north would also have good reason to emigrate.
So I suspect by the time the Southern Knigdom fell, there were a lot more than just members of the tribes of Judah, Benjamin and Levi living there.