Compete with them for jobs? The forced labor already accomplished that by locking people out of jobs and stifling the existence of a labor market. So, yes, they were impoverished in the South - and slavery helped to accomplish that.
Yet poor Southerners still saw themselves as being higher socially than blacks.