What Threatens Human Control of Military AI
https://www.usni.org/magazines/proceedings/2025/june/what-threatens-human-control-military-ai
The Department of Defense (DoD) has repeatedly voiced intention to maintain “appropriate human levels of judgment” over autonomous and AI-enabled weapon systems, but those who study frontier AI technologies have identified many challenges to aligning AI with human intentions. While questions of AI alignment occasionally smack of science fiction, frontier AI developers grapple with these challenges for product optimization and safety. Two challenges relevant to military AI applications have emerged recently: sycophancy and emergent misalignment
In the technical sense, sycophancy is the tendency of AI models to offer responses that are pleasing to their users at the expense of being truthful. The phenomenon of sycophancy is thought to result from reinforcement learning through human feedback. Researchers demonstrated a remarkable tendency to alter answers to conform to user beliefs and preferences even when the model appeared to hold knowledge of appropriate ground truth.
Fine-tuning is the process of retraining a previously trained model on a domain-specific dataset. It is normally intended to boost the model’s performance within that domain. But, in a 2025 study, researchers identified “emergent misalignment.” In this work, researchers fine-tuned GPT-4o and other models on harmful (but not quite malicious) tasks: writing insecure code, for example, by assigning inappropriate file permissions when copying files.8 The researchers found that, post tuning, the model not only wrote other poor code, but it also often exhibited a wide range of harmful behaviors on unrelated tasks and queries—including encouraging violence and glorifying Nazis.
Automated systems have advantages in their speed of decision-making, scalability, efficiency, and information storage capacity. But AI alignment challenges suggest that any human may find intent edited, eroded, or even eclipsed by operation of the human-machine system, creating an inexorable pull toward deference to the machine. Highly advanced systems will prompt the question: Who is aligning whom?
Same old GIGO.