They are only an inconvenience when they do not assimilate into the US culture / English language.
It's not only non-white immigrants who do not assimilate but that's never pointed out. Plenty of native born Americans do not assimilate into the 'culture' of the US. As for speaking english, well my mother speaks english and she is STILL told to go back to her country (although she's a naturalized citizen).