Well, if you make that arbitrary a moral axiom. Most would say, as an example, that a man who relentlessly goes about shooting and torturing and killing innocent little children does not have the same claim to self-defense as someone like St. Francis. But of course, they have moral axioms that set their own ideas of good and bad. Most people get those axioms from God, rather than arbitrarily making them up in their heads.
Who says? You? There is nothing in atheism from which to distill the existence of such a right. The Darwinist model of mindless, aimless, godless natrual selection and survival of the fittest posits no moral "truths" at all. The only truth it posits is: the stronger, cleverer, and more ruthless get to mate and leave offspring; the rest die. The "right" to self-defense is an abstraction. Darwinist theory gives no place to abstractions.