That you're even attempting to defend this is beyond belief.
Obviously variations of the English word damn are found in King James Version so, prima facie, it is not obscene. Neither did the phrase blaspheme. Neither was the phrase used a curse against you. It seems to me it was "expressing anger, surprise, or frustration."
KJV is the only translation I find using damned but the word is a verb...but then I haven't done an extensive study on the topic nor do I intend to.
Now, if we want to play your little game where we pull out "bad words" in the Bible we can.
However, as used in the phrase in question, "damn" is profanity.