This was supposedly one of the reasons for the Riots in New York.
The combination of the Emancipation Act, institution of a draft in which the wealthy could purchase substitutes, and the resentment of Irish immigrants to being treated more poorly and with less respect than free blacks (and a bit of encouragement by the local Democratic Party ... some things don’t change) all combined to foment the worst riots in American history.
And for the development of licensing and permitting schedules, along with unionization of the trades.