For you and me, that is indeed the difference. Problem is, the folks on the other side of the argument obviously think otherwise. That's one of the points I'm trying to make.
You're basically assuming that the other side shares your moral viewpoint on the matter, when in fact they do not. You've got to be clear on the grounds of the debate.
Finnly getting back to this - "the other side" are the people who believe morality is subjective, so there is no possibility of EVER winning a moral debate with them. Any time the final answer is "if it's wrong for you, don't do it, but I don't think it's wrong for me, so don't tell me not to do it" there's just nothing to debate, and the only arguments have to be pragmatic.