That’s true, but app 20 years ago a man was still hounded from his job for saying it, even though he said it in context.
Obviously the facts of the matter (the actual origin of the word "niggardly") don't matter--it's whether someone is offended because the word is perceived to be racist.