Your size and build reminds me of a member of UDT-14 I once knew. He was that way into BUD/S and even after he came out he did not look like he was well muscled but anyone who started something with him was in for severe trouble. Take a variety of martial arts course if for no other reason than the self control necessary to master any of the disciplines will provide a great deal of confidence. There is nobody who is invincible there are only men who are willing to put it all on the line and who are will to train to do the best possible job when forced into a fight.
In short you do not have to be the baddest dude on the planet just more willing to try than the other guy. A comprehensive approach to self defense is what will work.
In the final analysis that is the basic presumption behind CCW the person who carries a sidearm for self-defense will usually try to walk away from most fights and willonly fight when cornered and there is no other choice but when faced with that situation will survive by resorting to overwhelming force.
Stay well - Stay safe- Stay armed - Yorktown
That reminds me of Patrick Henry's speech when he spoke of the battle going not to the strong, but to the vigilant, the active, and the brave. I caught the last acronym but the first two UDT-14, BUD/S I'm unfamiliar with. The second is basic traning of some sort?