The union troops were segregated so no black regiments were there. The blacks were imbedded on Confederate Regiments, so they were there.
It makes no sense to keep your cooks and body servants and laborers and teamsters in separate regiments, does it? So of course the rebel army kept them close at hand to the white men they were servicing.