let them go. you can't use that argument to justify massive taxation of those who work, and low taxation of those who relax all day.
Low taxation of those who relax all day? They pay the same income taxes everyone else does. Sure, many of them don't pay SS or Medicare taxes, but they won't be using those services, either.
Unless you think they actually take more from the country (the public, not their accumulated wealth) than they contribute, wouldn't it just cost more to lose them?