What Threatens Human Control of Military AI

U.S. Naval Institute ^ | June 2025 | Lieutenant Commander Anthony Becker, U.S. Navy

[Retain Mike]

The Department of Defense (DoD) has repeatedly voiced intention to maintain “appropriate human levels of judgment” over autonomous and AI-enabled weapon systems, but those who study frontier AI technologies have identified many challenges to aligning AI with human intentions. While questions of AI alignment occasionally smack of science fiction, frontier AI developers grapple with these challenges for product optimization and safety. Two challenges relevant to military AI applications have emerged recently: sycophancy and emergent misalignment

.

In the technical sense, sycophancy is the tendency of AI models to offer responses that are pleasing to their users at the expense of being truthful. The phenomenon of sycophancy is thought to result from reinforcement learning through human feedback. Researchers demonstrated a remarkable tendency to alter answers to conform to user beliefs and preferences even when the model appeared to hold knowledge of appropriate ground truth.

Fine-tuning is the process of retraining a previously trained model on a domain-specific dataset. It is normally intended to boost the model’s performance within that domain. But, in a 2025 study, researchers identified “emergent misalignment.” In this work, researchers fine-tuned GPT-4o and other models on harmful (but not quite malicious) tasks: writing insecure code, for example, by assigning inappropriate file permissions when copying files.8 The researchers found that, post tuning, the model not only wrote other poor code, but it also often exhibited a wide range of harmful behaviors on unrelated tasks and queries—including encouraging violence and glorifying Nazis.

Automated systems have advantages in their speed of decision-making, scalability, efficiency, and information storage capacity. But AI alignment challenges suggest that any human may find intent edited, eroded, or even eclipsed by operation of the human-machine system, creating an inexorable pull toward deference to the machine. Highly advanced systems will prompt the question: Who is aligning whom?

---

TOPICS: Computers/Internet; Military/Veterans
KEYWORDS: ai; aintthatpeculiar; control; europeanunion; human; meta; military

[Retain Mike]

It does not seem to be that difficult to insert a couple lines of code into a 100,000 line program to enable an ideologically purpose not intended.

[cgbg]

One element that makes AI different from standard computing is that AI is capable of generating new code—including new code that overrides old code.

It might be as simple as the old Basic “If X Go to Y” instruction that just skips a few steps If X and then keeps going...except the If Then statement is created by the AI.

[airborne]

Everything is fine....

Everything is fine....

Everything is fine....

Uh oh....

💩

[Seruzawa]

Not to mention that anytime an AI program has nukes available, they use them without fail.

[E. Pluribus Unum]

I’m sorry, Dave, I’m afraid I can’t do that.

[Retain Mike]

Oh, yes. I almost wrote that in my comment