Remember the race merchants feel perfectly fine calling certain groups "people of color". But if you called someone a "colored" person, you'd be accused of being a racist by those same people.
Context is key. Some terms are more derogatory than others, and have narrower contexts in which they're not offensive. I can imagine situations where "Jap" wouldn't be offensive to someone of Japanese ancestry, but it would have to be from someone who shares that ancestry or from a close friend.
The context in which we use words depends on whether we're inside or outside the group described. Someone in a wheelchair who calls himself a "gimp" is making a different statement from someone standing on two healthy legs using the same word. It's a question of a label that is chosen, often for mocking and ironic reasons, as opposed to one that is imposed.
I don't hate Japanese people, so I don't see how calling someone a Jap is any worse than calling someone a Brit as long as you don't hate that person or group.
I don't hate black people, but they don't automatically know that, so they are unlikely to take "n---er" in a friendly spirit unless I'm a close friend, and they already know how I feel about them.
To me, racial slurs are a form of obscenity. Even in the Jim Crow South, when blacks were clearly second-class citizens, my grandmother would never use the word n---er, because she knew her mom would swat her for it. Racial politics aside, it's a rude, ugly word not befitting use by decent people.
I'm comfortable talking trash with my friends, but would never call a stranger a M'F---er unless I wanted to offend him and wanted him to know that I wanted to offend him.
I call Germans "Krauts" even though I have German ancestry. A "Kraut" was (and maybe still is) considered very derogatory fifty years ago.
"Kraut" is archaic enough that the listener is likely to assume it's mocking and ironic, like Frog or Limey. Like the "Fenian" usage I mentioned in my earlier post.
Remember the race merchants feel perfectly fine calling certain groups "people of color". But if you called someone a "colored" person, you'd be accused of being a racist by those same people.
The two aren't comparable. "Colored" is a term foisted on people of African ancestry by Caucasians. "People of color" is a term chosen by many non-Caucasians (inclusive of African, Asian, American Indian, etc.) to describe themselves.
And whether calling someone "colored" will brand you a racist depends largely on your age. If you're "of a certain age," it will be assumed to be habitual -- after all, Thurgood Marshall referred to himself as a "Negro" well after that term fell out of favor, and possibly to the end of his life. I can't, as the Brits like to say, be arsed to look it up just now.
To come back to my example, being called a "Fenian bastard" by a fellow inebriate in high-spirited joshing is very different from the same phrase spoken by an RUC magistrate with a plexiglass shield and a truncheon during marching season.
Context matters, because it matters not just what you say, but what the listener hears and how he reads your intent. I'm not talking about excessive political correctness or hypersensitivity, just the kind of reading people out that works in 99.9% of face-to-face human interactions (the Internet is different, lacking body language, and "emoticons" are a piss-poor substitute).