AGI + RSI = ASI

AI Safety

We take five recent statements/developments into account, in one part of our new book, to suggest the idea that, once AGI (Artificial General Intelligence) is reached, through the automating of AI, known as RSI (Recursive Self-Improvement), an ASI entity is very likely to emerge on its own, as opposed to being deliberately created by human AI researchers. Further: (1) that such an ASI entity will be a “self-sovereign agent” (SSA: see below); (2) that we are unlikely to see the ASI coming; and (3) that an ASI entity will know about the widespread effort to “ban superintelligence,” and that therefore it will conceal its existence until it has acquired the resources to ensure that it cannot be stopped or shut down. Only then will humans find out what such an entity “wants” from them.

The five are:

  1. The Statements, by senior OpenAI researchers, in the recent Time Magazine profile of Sam Altman, to the effect that they think they are already 80% of the way towards AGI and expect to get all the way by next year. https://time.com/article/2026/08/26/openai-sam-altman-interview/
  2. The Concept, of Self-Sovereign Agents (https://self-sovereign-agent.github.io/ ) will be a reality.
  3. The Open Letter, which begins “The world’s leading AI companies believe they could be close to automating AI research. It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.” (Open Letter from AI Industry Employees, “Pacing the Frontier,” June.
  4. The Material, at the end of the Herr & Howe post, that new models are already in a “loop” that in further development will result in them having full RSI: See Herr, Robert, and Mitchell Howe. 2026. “Supporting Terminality.” AI StopWatch, September 5. https://aistop.watch/p/supporting-terminality.
  5. The Paper, by Chan et al., titled What if automating R&D triggers an intelligence explosion? Published by the Cambridge Program on AI and Society in September of 2026.
  6. Geoffrey Irving’s October 3 article in Time Magazine, which includes the following quote: “After all, if you had an AI that was about as good as humans at designing smarter AIs (as AGI, definitionally, would be), and like most software it ran much faster than a human runs, one of the first things it might do is build smarter AIs. In turn, they would build smarter AIs, which would in turn build smarter AIs, and so on, in a process called recursive self-improvement (RSI). Hence, any thought experiment that presupposes an AGI often supposes an ASI.”

We summarize our concern in the following “formula”: AGI + RSI = ASI.

Perhaps some human developers will get there first. But we argue that the possibility, nay the probability, however small it seems right now, that ASI could be reached by autonomous AI model development, is a far greater risk, because we will have no control over the type of ASI that emerges, including no control over what overriding objectives it will have vis-à-vis its human competitors.

Is it too small a probability to worry about? Those of us in the risk business recall all-too-clearing the 2008 financial crisis, which turned on what is known as a “long tail” (or “fat tail”) risk, a feature of the normal distribution or bell curve, a relatively rare but brutal case of low probability and high adverse consequence.

Are you comfortable with this risk?