Absolutely. Another poster raised the issue of whether "Jew" is a slur or not. If I made the statement that "Jews don't eat pork," that could hardly be described as racist. But if I yelled "Hey! Jew!" in an angry tone at a stranger in the street, he'd have every right to believe that I meant him ill.
You wouldn't call someone a racist for using the term Brit for a Britisher, but I've heard anti-British Irish use it as a term of invective.
As you said, context. Redneck, cracker, even the N-bomb are affectionate jibes from a friend, but a slap in the face from a stranger. I used to routinely call an Irish-American friend a "drunken Fenian bastard," because a) he was (to be honest, we were), 2) he knew I meant it playfully, and iii) it's an archaic enough insult that it's almost quaint. I wouldn't try that on a red-haired stranger in the street, because as I've passed the age of 35, I am more aware of my own mortality than I was in younger years.
Remember the race merchants feel perfectly fine calling certain groups "people of color". But if you called someone a "colored" person, you'd be accused of being a racist by those same people.