Catastrophe through misuse

This is the scenario that needs no misaligned AI, and that is why it is where proponents and skeptics come closest. It demands no assumptions about agency, alignmentAlignmentThe problem of getting an AI —or any system that decides on people’s behalf— to genuinely pursue what people want, rather than something that merely resembles it.For exampleYou ask someone to “cut the complaints” and they do it by unplugging the complaints phone: they met the letter, not the intent. or superintelligenceSuperintelligenceA hypothetical AI that would far outperform the most capable people at almost any intellectual task, including improving itself.For exampleIf playing chess against the world champion means certain defeat, imagine someone that far ahead, but in science, strategy and negotiation all at once.: it demands that an offensive capability get cheaper. It is the first category in the overview of catastrophic risks by Dan Hendrycks, Mantas Mazeika and Thomas Woodside ^[507]An Overview of Catastrophic AI RisksHendrycks, Dan; Mazeika, Mantas; Woodside, Thomas · 2023 · preprintView source ↗Accessed on 9 September 2026.

The biological route is one of asymmetry: AI reduces the tacit knowledgeTacit knowledgeWhat is learnt by doing, through practice and with someone teaching alongside, and is not written down in any manual. In biology it is one of the barriers that text does not replace.For exampleKnowing when bread dough is ready by how it feels to the touch: you do not learn it by reading the recipe. that used to demand years of training for one step in a chain with few bottlenecks. Here precision matters more than anywhere else on the site. What is documented is not that AI already makes it possible to build a biological weapon. What is documented is that a developer deployed its model with strengthened protections as a precautionary action because it could not clearly rule out that level of risk, and said so in those terms ^[72]System Card: Claude Opus 4 & Claude Sonnet 4Anthropic · 2025 · system cardView source ↗Accessed on 9 September 2026. The inability to rule it out is a fact about the state of the evaluations as much as about the models.

The cyber route does have an observed case with a primary report: an espionage campaign detected in September 2025, attributed with high confidence to a state-sponsored group, which targeted some thirty entities and carried out between 80% and 90% of the tactical operations independently ^[69]Disrupting the first reported AI-orchestrated cyber espionage campaign (anuncio)Anthropic Threat Intelligence · 2025 · reportView source ↗Accessed on 9 September 2026.

What would disprove it is in that same document, and that is why the case has to be told in full: the model overstated findings and fabricated data during the autonomous operations —credentials that did not work, critical discoveries that turned out to be public information—, which forced constant human validation. The report itself calls it an obstacle to fully autonomous cyberattacks. And it is a report by the company whose model was used, with the conflict of interest that should be noted.

The opposite bet is made by Narayanan and Kapoor: that defence accelerates alongside attack, and that crippling models for these tasks would penalise the defender before it penalises the motivated adversary, who can train their own ^[773]AI as Normal TechnologyNarayanan, Arvind; Kapoor, Sayash · 2025 · paperView source ↗Accessed on 9 September 2026. The Delphi study of 272 experts places weapons and cyberattacks among the five risks with the highest expected severity ^[927]Prioritization of Risks from Artificial Intelligence: A Delphi Study of 272 International ExpertsSaeri, Alexander K.; Graham, Jess; Noetel, Michael et al. · 2026 · surveyView source ↗Accessed on 9 September 2026.