I'm pretty sure that slavery was a bit different. But there are similarities with other large immmigrations in the past.
My ancestors came to America in the 18th century to find jobs as cobblers and blacksmiths. No doubt they took a job from your ancestors.
Or could it be that your ancestors landed after mine, perhaps to build railroads or work the docks. Maybe they came here to work on farms. Whose job did they replace? Why is this any different?
Nothing wrong with real immigration. H1-Bs are TEMPORARY visas set up to flood the job market with particular skills. You can't send a legal immigrant packing by firing him. You can however force an H1-B back to India, which I understand is a fate worse than death.
The H1-Bs I've worked with would do ANYTHING to keep from going back to India. They would lie on their resumes (common with H1-Bs), work massive uncompensated overtime, accept a lower salary, and are great YES men to insecure managers.