But others do not have to afford it to you.
Indeed they do, if they wish to rationally claim it as a moral imnperative for themselves.
Those who are willing to initiate force against otherwise peaceful men, rationally preclude any claim they might make to such a moral imperative.
Have many done so anyway?
Sure.
But their actions cannot be rationally defended or sustained consistent with the notion of morality.
Hence, they act immorally.