Your assumption that with emancipation, whites would naturally hire other whites over the freedmen, is not borne out by the survey which the chief Actuary for the Prudential Insurance Company conducted on the subject in 1895. He found much the attitude to which Booker T. Washington appealed in the same era--that there was if anything a sentimental bond between the more affluent whites and their ex slaves, and that the poor whites were seen as much more difficult to get along with--less desirable as employees.
You may not want to admit that it was the egalitarian rant of the Reconstruction demagogues, which undermined Negro society; but we have seen exactly the same process repeated with respect to the egalitarian rant of the "Civil Rights" movement. Just look at what has happened to crime and illegitimacy statistics, contemporaneous with the imagined "gains" from what was and is, inherently, a Socialist movement.
You cannot destroy the cultural images of a people and substitute leftwing fantasies without taking a terrible toll. (I must sound like a broken record, but we address the images of the old South--the images on which people could have built a better life for both races in the 20th Century--in The Persuasive Use Of Images.)
William Flax
But the racial fears that you refered to led Whites to bar Blacks from the franchise. Once that was done, rich Whites had to give work to poor Whites to keep up their own power in the community, however much they might have esteemed Blacks as workers. White fears of race-mixing and of being outnumbered by Blacks were in evidence even under slavery, well before Reconstruction, so it's not clear how much radical Reconstruction had to do with later race-relations. Radical Reconstruction could be pointed to as a justification, but all the reasons for segregation and the consequent decline of Black craftsmen were already in evidence before.