You're not. But there are also many who understand that history doesn't stand still and phrases that were once meant in a racist context can be coopted by the larger culture and stripped of their original racist meaning. That certainly hasn't happened with many or most stereotypes or phrases but I believe it has happened with this particular one.
Here's your test - do me a favor go to a black club walk in and shout out the phrase - I will promise it will get very uncomfortable.