In fact, black units were raised almost from the beginning of the war. The first known combat by a black unit was against the Confederates on October 27 & 28, 1862 at Island Mound, Missouri by the 1st Kansas Colored Volunteer Regiment before being mustered into service. The regiment was organized in August 1862, mustered into service January 13, 1863, and later redesignated the 79th United States Colored Infantry Regiment. What you are confusing are the USCT regiments, the formation of which began in 1863, and state volunteer regiments. Those were formed as early as September 1862.
According to your own numbers there were 4.5 million slaves in the South!! And even accepting your number only 90,000 fought for the north? And reread that page. Those folks were in occupied Southern territory meaning they didn't escape north, the northern invaders came to them. Then not even to free them but to maintain status quo and 'hey while you're at it fight for us and we'll feed you since we've about burned or destroyed everything else around here'. What do you think the other 4,410,000 or so were doing? They couldn't have all stayed at home. No, because just as many blacks were fighting or supporting the South in one form or another.
As for the 180,000 number, it's not mine, it's the city of Alexandria VA with a big ol' picture of Frederick Douglas. You think it would pass the Asa Gordon test? I don't even know that it's true, but I figured hey if this government authority puts up a picture of Douglass not even recognizing his own statement about the blacks fighting for the South, it has to be liberal enough for y'all to accept it