You are not the first to propose this. See Heinlein's "Beyond This Horizon" as an example:
One sub-theme of the book is the carrying and use of firearms. In the novel being armed is part of being a man; otherwise he wears a brassard and is considered weak and inferior. Women are allowed but not expected to be armed. Duels, either deadly or survivable, may easily occur when someone feels that they have been wronged or insulted, a custom that keeps order and politeness. A defining quote from the book, "An armed society is a polite society", is very popular with those in the United States who support the personal right to bear arms. (cited from here.)
It is also curious that the book is part of the "early, liberal-minded Heinlein" period (the novel was first published in 1942.) I guess the meaning of many words changed in last decades.
Not all who wore the Brassard of Peace were weak and inferior. I must note that the “Guardians” (IIRC) who were the Police Officers of that society wore both the Brassard of Peace and carried arms. Everyone was thereby put on notice - by the combination of the Brassard and the weapon - that they could not be fired upon.