A fair question.
The answer goes back to the time when ship's had wheels and smaller vessels had tillers. To turn a ship clockwise you turn the ship's wheel in the same direction. To turn a small boat clockwise you move the tiller in the opposite direction. Using port and starboard could introduce the possibility of error which is unacceptable.
Orders to the helm have VERY specific meanings which the helmsman will carry out EXACTLY. Part of learning to be an OOD is learning, and internalizing, those commands so that you will instinctively give the correct orders so that the ship maneuvers as you intend.
WWG1WGA
Garde la Foi, mes amis! Nous nous sommes les sauveurs de la République! Maintenant et Toujours!
(Keep the Faith, my friends! We are the saviors of the Republic! Now and Forever!)
LonePalm, le Républicain du verre cassé (The Broken Glass Republican)