Slaves were offered their freedom in return for signing up. See Jordan, "Afro-Yankees and Black Confederates in Civil War Virginia," Virginia University Press. Moreover, there were some---though not many---"free men of color" still in Virginia on the eve of the Civil War who could sign up.
BUT those FEW were the EXCEPTION rather than the rule.
free dixie,sw