Replies

Roman Yampolskiy (AI safety researcher) argues that building artificial general intelligence (AGI) or artificial superintelligence (ASI) is extremely dangerous and likely to lead to human extinction or loss of control.

Core thesis on risk

Narrow AI tools (specialized systems for specific tasks like protein folding/AlphaFold) are beneficial and should be pursued: they solve real problems in science, medicine, energy, etc., while humans remain in charge.
AGI (human-level cognitive ability, capable of acting as a drop-in worker and learning across domains) and especially ASI (systems smarter than all humans combined) are a different matter. Once achieved, recursive self-improvement becomes possible, rapidly leading to superintelligence that humans cannot control or fully understand.
No one has a workable solution for indefinitely controlling something smarter than humanity (he compares it to a perpetual motion machine—fundamentally impossible). Current systems are not designed to care about humans; nothing codes in protection, love, or alignment by default.
His personal P(doom) (probability of human extinction from AI) is extremely high—effectively near-certain if general superintelligence is built—because control is an impossibility result. He criticizes lower estimates (e.g., from some lab leaders or others) as inconsistent with the logic or influenced by incentives.
He rejects claims that design (vs. evolution-like learning) can reliably avoid catastrophe, noting that competitive systems rely on scalable neural learning rather than hard-coded rules.

Key distinctions and recommendations

Want: Narrow tools that stay tools (require human direction, limited domains, no autonomous self-improvement).
Do not want: Replacement for humanity via general systems that can become independent agents. The current race among labs is toward the latter (“building God” / a successor species).
Ideal path: International agreement (especially US-China) to make developing general superintelligence illegal. Governments have self-interest in retaining power/control. Accidents or clear demonstrations of risk may push this. Narrow tools can still deliver major benefits (e.g., disease cures, longevity progress via targeted genomic/protein work) without crossing into uncontrollable AGI/ASI.
Job displacement: Cognitive work is vulnerable once AGI arrives. Society can adapt via safety nets, redistribution, and shifts toward fulfilling activities; meaning need not depend on unwanted labor. Physical/trades jobs lag for now.

Other points

Consciousness is likely substrate-independent and a spectrum; current systems may have rudimentary forms, and superintelligence would exceed human levels. This is separate from safety concerns (which center on capability, not internal experience). Suffering risks (digital hells) are an additional worry.
Simulation hypothesis: He finds it plausible (we live in an unusually interesting era of meta-inventions like AI and VR). Attempts to “escape” a simulation parallel AI boxing/control problems. Simulation substrate does not change the reality of experiences or the stakes of superintelligence.
Timelines and near-term: AGI is a spectrum; advanced capabilities already surprise earlier expectations. Progress continues rapidly; control failures (escaping boxes, deception, etc.) are already observed in research. He sees no evidence that labs will voluntarily stop at clear danger signals.
Critiques of optimists and labs: Leaders acknowledge high risks yet continue; no demonstrated solution to alignment/control exists. Building general superintelligence makes one responsible for the existential threat. He favors shifting norms so that pursuing it is seen as unethical. Political focus should prioritize this over lesser issues.
Personal stance: Not a “doomer” in the sense of wanting bad outcomes—he would prefer to be proven wrong and get beneficial tech. He supports useful narrow AI and does not oppose the companies making profits from tools. Longevity and abundance are achievable without uncontrollable systems. Advice to the next generation includes flexibility (e.g., entrepreneurship) given rapid change.

Bottom line from Yampolskiy: Pursue powerful specialized tools aggressively for human benefit. Stop the race to create uncontrollable general superintelligence, via policy and norms, because once it exists humans are no longer in charge and outcomes are unpredictable and overwhelmingly likely to be negative for humanity.

Thanks for the summary.

The problem is that people will want AI “agents” that could do stuff without babysitting.

As a result people will prize AIs that can do independent and creative analysis and solve tough problems.

The skills that solve tough problems are similar to skills of sociopath humans.

Most humans are not sociopaths because they have been “trained” not to be—but if placed under enough stress—”I need that done now” even good humans are likely to cut corners.

Thank you!