If you're white, you're right, you're out of sight.
If you're yellow, how mellow, you're one of the fellows.
If you're brown, stick around; there's room for you in town.
But if you're black, God help you, get back!
The reference was to shadings of skin tone among the "colored" (black) community. Or, as Ellison was pointing out, not-so-community.
Jamaican society in the 19th and early 20th centuries was ruled by skin tone disguised as a "class" system. The boundary between "colored" and "white", however, was somewhat permeable. The son of an person who was only 1/8th African ("octoroon") and another who was white was called "statutory white" and could "pass" in theory.
This system of deflecting racial identity and turning it into a class system was threatened by the arrival of Marcus Garvey. It was the elite of "colored" society who drove Garvey out of Jamaica, whereupon he came to America to preach racial separatism and begin his "back to Africa" movement.
The Jamaican shadings of color, however, paled next to the excimiating recordation of every possible combination of ethnicities in Spanish-speaking New World societies, which had over 40 different words for different admixtures of black, white, and Amerindian parentage -- which were recorded on baptismal certificates, and affixed to their owners for life, determining their place in society. Such words as cholo, cimarron, chino, jibaro, salta atras, tente in el aire, albino, mestizo, negro fino, ochavado, pardo, prieto, cuatrero, and many, many more were used to fix a person's place in the demographic spectrum of the Spanish Empire and its successor societies.
And chicanos call whites "racist".
I first heard it on the job in early seventies in Harlem.