Realistically there needs to be a security force rating. IIRC this was supposed to be in transition back in 1980's with sailors assuming some of the security duties and Master At Arms being made a permanent rate beginning at E-5 or so. Obviously something fell through.
So who should be well trained in arms including small arms? IMO those most exposed. A good portion of the Bosunmates for example should be trained for several reasons. They handle the boats and there has been cases where liberty launches have come under attack. All Weapons Dept should be trained as well. Each ship needs a sufficient onboard duty section trained to secure the ship. Obviously the smaller the ship the more cross training and qualifications will be needed and a stand alone dept not practical.
A security team can be done and I can give an example of it. Back up till 1980 on my ship at sea fires below the flight deck were fought by Hull Technicians. The were roughly 90% of the fire party. In 1980 a permanent onboard full time Fire Department was put in place while in the ship yards and made permanent afterward. They also took on maintenance of fire stations, repair locker, hatches as a secondary function. It was a pooled Department from all departments.
The concept addressed several issues. More persons were trained to fight fires and if a major fire occurred the ships Hull Techs would not perish in first response situation. I put in nine months there myself three months in the 78 yard period and six months during our 1980 one year overhaul. The second time I was a nozzle man and had gone through at least one major fire in my prior stint. It worked a lot better in port also because the duty sections had daily trained together and had sharpened skills. Before that if you were DC qualified you might be on in port FD one duty day and not on it the next if ever again. The same concept can apply for security but only if providing security is the primary function and proper training is given.
Not trying to take anything away from Sailors, my Dad was a Sailor WWII, my Son Was a Sailor, Desert Storm....but are there no longer Marines aboard U.S. Navy Ships ?