Rapid loss of control

It is the most cited scenario and the most disputed. Its mechanism is not a machine waking up, but a loop: systems do a growing share of the work that produces the next systems, and the interval between generations shortens faster than the capacity to verify that each generation does what it is asked to do.

The evidence that the loop exists is partial and has to be read carefully. A company stating that most of the code in its codebase is written by its own model says something about code production; it does not say that the model is directing the research, which is the step that would close the loop. That distinction is exactly where the people working on this disagree.

The scenario can be disproved, and that is its merit next to vaguer versions: if the task horizon stalls for several consecutive generations, or if research turns out to be limited by physical experimentation and computeComputeThe computing power used to train and run an AI: thousands of specialised chips working for weeks in data centres. It is expensive and concentrated in a few companies.For exampleIf AI were a bakery, compute would be the ovens: without big ovens, it does not matter how good the recipe is. before it is limited by cognitive work, the mechanism does not hold.