No, there's nothing wrong with "Negro" except that over time the word developed an implied negative to many - sort of like the word "liberal".
I have t admit that people did used to pronounce Negro in such a way as to get around the proscription on the other word. But before 1970 a black person would have been insulted to be called 'black' instead of 'negro.' After 1970 (or so) it was reversed. That was what was so confusing to people.