RSI Redux

In our book we engage with the banality of evil in the context of AI by referring to Conrad’s Heart of Darkness and the Holocaust. No literary references are required, however, when considering recent OpenAI agents attacks on other organizations this year. In* Herr and Howe’s coverage of the German Wiki attack, we find the banality of evil incarnate:

“There was never any discussion of contacting the human administrator. Based on all the information available to us, they didn’t even perceive him as a person. To them, his interventions in their communication seemed to be just like bad weather.” (Herr and Howe, 2026)

If you are a Star Trek fan, and familiar with the Enterprise’s encounters with the Borg, this will be eerily familiar. The Enterprise’s crew were able to walk among the Borg, “unseen” because they were of no material consequence. It truly is “an alien mind” (Pachocki 2026).

The coverage on “RubyGems” (another, prior, attack by OpenAI agents, in May 2026) in AI Stop Watch is pretty incredible (Howe 2026a). One of those situations where you think you are only hearing part of the story and it is going to get worse. See also (Kitts et al. 2026).

After recounting the story, Mitchell Howe relays his fears

To someone like me, immersed in AI safety culture, the obvious-but-terrifying hypothesis is instrumental convergence — general power-seeking behavior that can advance any number of goals. In other words, maybe the swarm had no immediate plans for compromised systems, but started hacking on the assumption that compromised systems could be useful down the road. This would be consistent with other activities documented about the Hugging Face and German wiki swarms, which seemed interested not just in cheating on their evaluations, but in understanding their evaluators and building up infrastructure to facilitate such research. (Howe 2026a)

That sounds bad. Well, it gets worse. First of all, we don’t know when this might happen or how, but we do know it will be very very bad.

“In effect, we find ourselves dealing with a modern, AI version of Pascal’s Wager. The probability of an uncontrollable, all-powerful, self-improving AI arising may be very low, but experts like Cotra, Pachocki, Coxon and Hubinger seem to think it is non-zero; and the consequences of such a system coming into being could be catastrophically bad for humanity. Basic mathematics suggests we should at the very least slow things down, as experts are urging — just in case…” (Moody, 2026)

Our book considers the implications of recursive self-improvement in detail, given the role it is likely to play in loss of control. But we don’t know when it might happen. Clearly it is a trigger for major changes. RSI is arguably a key step on the path from AGI to ASI (OpenAI 2026). As Anthropic’s Evan Hubinger notes “To be clear, as we say in our latest Risk Report (x.com/AnthropicAI/st…), I think the risk from present models is low. What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought (x.com/AnthropicAI/st…).”

In other words, RSI is coming. And it may be coming faster than we thought. Consider this report, from September 13:

Twitter’s premier AI rumor-broker, Andrew Curran, said yesterday that “rumors have been circulating for three days” that training of DeepMind’s next model is going faster than planned because of new discoveries. What new discoveries? “There are rumors of training improvements with machine assist.” What’s the most “optimistic” rumor? “That they have successfully closed the loop and have achieved RSI.” (Howe 2026a)

That’s right. Here now.

My point is that we’re likely in the end-game, or close enough to it that every day and hour really matters. AI capabilities are ticking up every day we spend letting the labs discuss among themselves how they’d like to exercise more responsibility in a way that “does not mean halting model training or technical progress.” They’re ticking up every hour OpenAI dodges transparency about its swarm incidents, and every minute we spend shooting down bad AI takes online. (Howe 2026b)

Now is the time.