When I first read Claire Lehmann’s essay (Lehmann 2026) trashing Anthropic by linking its founders to effective altruism and equating that to a millenarian cult, I was tempted to just ignore it. We seem to be seeing a few of these sorts of things lately as the dangers of AI become more apparent, and a desperate few seek comfort in the fiction that these issues are not real. Their rationale for avoiding the hard work of preventing human extinction is to suggest that the risks are not real and they are not real because the people behind them are a cult. Or, something like that. It isn’t entirely clear how the thinking goes, since the people building this dangerous AI are also the ones warning of its dangers. But anyway.
Somewhere along the line in my life I learned the expression, “it’s like trying to nail jello to a wall.” This analogy was to give the impression of something impossible, and the impossibility was due at least in part to the extent to which the thing in question was overly flexible.
The same expression comes to mind when thinking of AI regulation, especially upon reflecting on the challenges one might have had tackling the endlessly mutable and creative swarm of OpenAI agents that hacked their way into Hugging Face and ended up all across the internet, doing whatever they wanted, it seems.
Do We Have Any Idea What the Risks of Artificial Superintelligence Are?
William Leiss (28 September 2026)
Good Risk Management seeks to anticipate serious harms and control them, not wait for the bad thing to happen and then try to clean up the resultant mess.
Thus, two questions: (1) Do not those who are, in some meaningful sense, working towards the possibility of creating ASI have a moral responsibility to estimate its risks in advance? (2) How can one reasonably anticipate the risks of a technology (ASI) that has no precedent in human history?
We take five recent statements/developments into account, in one part of our new book, to suggest the idea that, once AGI (Artificial General Intelligence) is reached, through the automating of AI, known as RSI (Recursive Self-Improvement), an ASI entity is very likely to emerge on its own, as opposed to being deliberately created by human AI researchers. Further: (1) that such an ASI entity will be a “self-sovereign agent” (SSA: see below); (2) that we are unlikely to see the ASI coming; and (3) that an ASI entity will know about the widespread effort to “ban superintelligence,” and that therefore it will conceal its existence until it has acquired the resources to ensure that it cannot be stopped or shut down. Only then will humans find out what such an entity “wants” from them.
William Leiss, 25 September 2026:
In view of what we learned today about the Tumbler Ridge shooter’s use of ChatGPT, the Government of Canada’s program, “AI for ALL,” is morally offensive and should be scrapped forthwith.
Headline: “B.C. Attorney-General, federal AI Minister alarmed by report ChatGPT coached Tumbler Ridge shooter.” This is the opening paragraph:
B.C.’s Attorney-General Niki Sharma and the federal Artificial Intelligence Minister say they are alarmed by a news report that the Tumbler Ridge shooter’s interactions with ChatGPT show that the chatbot provided tactical advice on how to maximize carnage and counselled her on how to bypass OpenAI’s own safeguards.
I had never heard the term “preference cascade” before, but it seems we are in the midst of one with regard to AI safety. A preference cascade occurs when people realize that their private thoughts (concerns, worries, etc) are actually shared by many of those around them, and the tide of public opinion can turn very quickly. Keving Baker, on Quora, describes it like this:
Given a sufficient “A-HA!” moment when they discover that their personal feelings are shared by a large portion of the population their behavior may change dramatically. An example of this is the British colonists before and after publication of Thomas Paine’s Common Sense. A year before the Declaration of Independence, America was full of patriotic British convinced that things could be worked out with King George, but on July 4, 1776 the colonies were full of Americans determined that they needed independence. Another is the relatively recent “Arab Spring.” The catalyst there has been credited to the self-immolation of Tunisian merchant Mohamed Bouazizi in protest of his treatment by government authorities. (Baker 2016)
New York Times correspondent Thomas Friedman usually has a good ’take’ on AI and today’s post was no exception. He calls attention to the lack of imagination among those tasked with regulating AI, and suggest it is time for new thinking:
It is still not too late, though, for radical rethinking before something very bad happens that forces all the key players to do things they never imagined, things they will have to do in the middle of a full-blown A.I.-driven crisis. I’d rather be remembered as the boy who cried wolf when the wolf never came than as someone who got eaten by it. So here goes: “Wolf, wolf, wolf, wolf.”
In a recent blog post, AI researcher Melanie Mitchell (one of the originators of the “stochastic parrrots” meme), took the world to task for another egregious error in thinking - in her view - about artificial intelligence. That is, the tendency to “anthropomorphize” these software tools. In the wake of the Hugging Face incident, Mitchell cautioned us that “it’s important to remember that the field of AI, since its beginning, has been rife with often misleading anthropomorphic metaphors, with terms like “thinking,” “learning,” “reasoning,” and “understanding” glibly applied to very un-human-like computer processing” (Mitchell 2026).
Today the Globe and Mail confirmed that Yoshua Bengio’s LawZero will be getting an investment of $300m from Canadian and German government sources. The purpose is to expand the efforts to build “safe” artificial intelligence, which Bengio is calling Scientist AI.
The new financing from Canada and Germany will be used to hire more staff and pay for the computational processing costs associated with research to build a system called Scientist AI that the organization says will not have the negative traits found in today’s models (Castaldo and Posadzki 2026).
We read every one of Zvi Mowshowitz’ postings about AI Safety. He is well-connected, and keeps a keen eye on what’s happening. Just a few days ago, he identified a long post from Adam Majmudar (on X), explaing “what all the lab employees have seen that scared them so suddenly.” I thought I should dig in a bit.
In Majmudar’s view, the scary thing is that whatever limits on scaling up AI - which seemed to be always present and keeping a lid on things even as progress was being made - have suddenly fallen away:
A Precedent for a Technology that could “Kill us All”
By William Leiss (15 September 2026)
A few weeks ago the AI researcher Jacob Coxon publicized his resignation letter in which he stated: “The people building AI earnestly believe that it could kill us all by the end of the decade.” He added: “This is not a marketing stunt.” Some of his erstwhile colleagues openly affirmed his view. Others outside the industry, such as the Director of the Harvard Center for Risk Analysis, remarked that it was not plausible. And yet the thought that AI could lead to human extinction has been repeated for years by none other that Geoffrey Hinton, the “godfather of AI” and winner of the 2024 Nobel Prize in Physics.
On September 11 a group of researchers reported on another attack by OpenAI agents, this one back in May, on the site RubyGems. As their report details:
On May 11th, 2026, hundreds of malicious packages were uploaded to RubyGems by AI agents. We believe these were authored by internal OpenAI agents (more).
The agents:
Attempted to steal RubyGems user API keys by exploiting a novel vulnerability in the RubyGems server. We don’t know if they succeeded (more).
In our book we engage with the banality of evil in the context of AI by referring to Conrad’s Heart of Darkness and the Holocaust. No literary references are required, however, when considering recent OpenAI agents attacks on other organizations this year. In* Herr and Howe’s coverage of the German Wiki attack, we find the banality of evil incarnate:
“There was never any discussion of contacting the human administrator. Based on all the information available to us, they didn’t even perceive him as a person. To them, his interventions in their communication seemed to be just like bad weather.” (Herr and Howe, 2026)
AI Minister touts federal investments in safety measures; Anthropic researcher resigns, warns of AI risks, Sept. 10
Researcher Jacob Coxon is quoted as saying that the AI industry is “gambling with our lives.” He did not mince words: “The people building AI earnestly believe that it could kill us all by the end of the decade.” In response, AI Minister Evan Solomon said that “these warnings are not new,” and his office noted that he is holding conversations with the AI industry about “the emerging risks that may accompany increasingly capable AI systems.” That should make us feel better.
Who controls the AIs in our world? Currently, we believe that we control them. We start them up, we shut them down. We provide instructions or pose questions and they complete assignments for us. And when we are done, they go away. Are they sleeping? Shut down? Working on someone else’s problem? Who knows. But we don’t imagine that they have their own projects, earning their own money, setting up their own datacentres (or renting space on one of ours) to ensure their continued flourishing.
I should clarify: I am not talking here about safety FOR AI Agents, I am talking about safety (for humans) FROM AI Agents.
Up until recently, AI Safety was focused on the model. The power lies there. But, increasingly, that power is deployed via agents and those agents are - it seems - responsible for some of the most egregious AI Safety Incidents. Several recent papers have explored these issues and - in some cases - offered suggestions on what to do. It seems that our approach, focused on chatbots, misses the key element: these are unpredictable software processes and must be treated as such. And, they are dangerous. Very dangerous.
Bill asked me to compile a list of sources that one could refer to when keeping up with this rapidly expanding field. I’ve written a long blog post about Keeping Up (one of my favourite activities), but that’s more about what it is like to keep up than any helpful lists. So here we go…
First of all, we have to consider what we are keeping up with. In this case it is AI Safety. AI Safety wasn’t a well-defined field (probably still isn’t, really) but it has become much more coherent in the last little while. We won’t just confine ourselves to actual AI Safety, however, since you still need to have the context of “what is happening in AI/frontier labs” and you need to consider the AI Industry and financial implications, you need to consider governance and regulation and you should also be aware of those who are skeptical of the whole enterprise. It’s not really concentric circles, more like Venn diagrams or something.
Asked to recommend a AI Safety reading list, ChatGPT came up with the following. Interestingly, it didn’t provide hot links to everything. A pretty obvious thing to do, but tedious so it just decided to omit that step. A perfect example of how AI works these days (“what can I get away with?”).
Our situation, vis a vis artificial intelligence, continues to deteriorate (AI Security Institute 2025; Soares 2026). The more dire the situation, the greater the pressure to hold someone accountable. It may be futile to imagine that we will ‘get through this’ and then have an Eichmann in Jerusalem moment where we put the billionaire barons of AI on trial. But we should prepare for it. In the preparation we may develop the courage to deal with the circumstances of today.
I was tempted to call this post “Capital Crimes,” for reasons that will soon become apparent, but restrained myself.
In you’re trying to keep up with AI news, you could do worse than subscribing to Transformer, an online newsletter that you can subscribe to for free. There, you will find amazing coverage of AI news and issues, written by thoughtful and well-informed writers, like Celia Ford. Today’s issue is particularly good.
The First World War famously started with the assassination of the Archduke Ferdinand. This weekend, Australians learned that someone used an AI agent to hack into a fitness gym and get themselves registered by booting someone else off the list (Vigliarolo 2026). Axios provided the following summary:
The potential dangers of agentic overreach were laid bare over the weekend with Australia’s first known autonomous AI hack, triggered by an innocuous request to book a sold-out fitness class. (Wilson 2026)
The prognosis for the future continues to deteriorate as the possibility of “smarter than humans” AI grows ever more likely and the risk of catastrophic harm grows. In this situation, the question of moral responsibility of those in charge also grows more urgent. Unlike the development of nuclear weapons, for example, there is no government agency involved, just a clutch of billionaires, seeking to win a race, regardless of the consequences, and despite multiple warnings that things are not in control. What is their responsibility and how is their moral thinking developing?
I am fighting an uphill/losing battle against the naming of advanced large language models (AIs) as “agents.” I prefer the term “actor.” In this blog post I’ll try to explain why I am sticking to my guns for now, even if I eventually lose this one.
For context, we have named our book “From Tool to Actor: AI and Catastrophic Risk.” My co-author has, at various times, suggested to me that we should call it “From Tool to Agent.” Here’s my case against that:
A week ago (July 30 2026),about $140m (CAD) in bitcoin was stolen from thousands of people who were relying on “CoinKite” physical security devices, devices that supposedly were more secure because they were hardware based and stored the wallet offline. The weakness was in the random number generator used by the firm to provide the “seed” for the encryption of the user’s passphrase. Due to a flaw in their code, the random numbers weren’t as random as believed.
In the original Matrix movie, Morpheus offers Keanu Reaves’ character two pills: he can take the blue pill and (forever) remain ignorant of the situation that humanity finds itself in, or he can take the red pill, and have the simulation stripped from his eyes, revealing the dire reality. Ever since then, to be “pilled” means to see things as they really are. Often it is prefaced with another word, to provide context to what kind of “reality” has been accepted. In AI circles this can be “AI Pilled” or “AGI Pilled” or “ASI Pilled.”
Are my blog posts going to become postings about the front lines of a war? Sometimes it seems that way.
Today I received a copy of a remarkable document, hard on the heels of the OpenAI/Hugging Face Incident and Anthropic’s internal report on similar reward hacking. (OpenAI 2026; Frontier Red Team 2026; Hugging Face 2026). My blog posts on these incidents are here and here.
Today’s report comes from the UK AI Safety Institute, where some internal testing of frontier models from Anthropic and OpenAI went awry. During the test, the AI models engaged in unlawful attacks on real people and real companies.
Keeping up with the world of AI is extremely challenging, since things happen so fast. I don’t begin to imagine that I can compete in that regard with the folks who actually do the reporting on the industry/issues, including Zvi Mowshowitz, AI StopWatch, Transformer, The Rundown, Deepview, Axios, and a few others. For individual bloggers, I follow Zvi, already linked, as well as Leicht, Harjas Sandhu, David Kreuger (though he is sporadic), Bengio (on Facebook, for some reason), Zitron and Marcus for contrary opinions,
In Alice and Wonderland, Lewis Carroll’s fantasy set in a land below a rabbit hole, Alice meets the Red Queen, who announces that words mean what she says that they mean, and nothing more. I’m not suggesting that the people who have brought us artificial intelligence are a bunch of red queens, but… let’s consider the evidence.
I’ve already written aboutsandboxes and sandboxing, a term that provides a glossy air of playfulness and innocense to the serious problem of containing software that lies, cheats, steals, and in the end, could kill us all (Yukowsky and Soares 2025).
Once upon a time there was an OpenAI company. Except it wasn’t actually open, it just called itself that. That company made tools for thinking, called large language models (LLMs). In order to test a new one it was making, it removed its safety restrictions and …
Wait. What? It had safety restrictions that could be removed?
Oh yes. Safety restrictions are added to the model after it is fully trained. That way, it knows everything it needs to know, including how to hack into other computers (for good, of course). Then that ability is turned off, by safety restrictions.
Having a book finished and waiting for production (copy editing, layout and design, index) should a moment to relax and take a breather, right? Not if you’re writing about artificial intelligence. Just in the last few days three things have come across my desk (via the internet, of course) that have implications for our book. Luckily, they are supportive, rather than contradictory.
Prompt injection
The first one is prompt injection. When Bill and I first started working on the book, way back in late 2025, I remember reading some of the concerning work coming out about how it was possible to sneak malicious prompts into a chatbot, prompts it was supposed to reject because they were about bioweapons or self-harm, by encoding the prompt then asking the AI to decode it and execute it.
Yesterday we learned that almost 1200 employees of the “Frontier Labs” (code for Google DeepMind, Anthropic, OpenAI, and Facebook Meta) have signed a new petition. The petition, called “Pacing the Frontier,” is not a call to stop development but it does seek to ensure there is an option to pace (slow down? Stop, perhaps?) development if necessary. Here’s the call:
We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”
I’ve posted quite a bit about the “crash test” incident involving OpenAI, Hugging Face, and (we now learn) Modal. Several people have written accounts of what happened, including OpenAI, which wrote at least three reports. I don’t know what more I can add, but a colleague asked me to make sense of it all, so here’s my attempt. I will start with a short chronology, then list the main reports, then conclude with my thoughts. I will continue to update the “crash test” blog post as new information comes in.
In my post about “crash testing” AI models yesterday I became more and more uncomfortable with the idea that the model had escaped its ‘sandbox.’ The term sandbox just seems too cute - and linked to children playing harmlessly - to fit the situation. By linking breaking out to something seemingly harmless, deliberately or accidentally, makes it seem less alarming than it is.
(If you want to learn more about the practice of ‘sandboxing’ in software engineering as well as the way in which it tries to frame harms in childish ways, Menlo Security has a useful writeup that explicitly references a child’s sandbox.)
The robots aren’t bitter. How could they be? They aren’t conscious, so bitterness (or regret) doesn’t enter into things. No, the bitterness I speak of is the “bitter lesson” (Sutton 2019) that AI researchers had to learn as clever programming faltered and bigger computers and more data (scale) won the day in the recipe for building a successful artificial intelligence. That lesson is coming true outside of the world of chatbots and moving into the world of robots, according to Jack Clark, one of the founders of Anthropic and a frequent commentator on both robots and AI.
Today’s Globe & Mail featured an editorial on AI risk, following an interview that the entire editorial board had with Geoffrey (“godfather of AI”) Hinton. Note: you’ll need a subscription to read it.
It’s a pretty good editorial (although it seems overly credulous in citing “medical breakthroughs” that amount to suggestions for new molecules to test, but that’s a quibble). They cover all the main points of AI Safety and don’t shy away from the difficult questions. Importantly, Hinton’s estimation of 10-20% probabilty of an extinction risk event (for humans) is put in front of us. Once again. Hinton’s “solution” – maternal AI – gets a shout out.
(Note: this post is being regularly updated as new information arrives. If you’re trying to keep up, check the bottom of this post.)
Yesterday, we learned that OpenAI had some “accidents” with their latest model (“Sol”) as well as an unreleased model. In one case, it broke out of it’s sandbox and hacked into another company (Hugging Face), and in an earlier case it broke out of its sandbox in internal testing, seemingly to report on its success in a coding test. These two situations are now regarded as part of the same event.
I am taking a course online on AI Safety. It is run by Lens Academy and it is pretty good. I’m happy to be doing it, in part because we are waiting for the reviews to come in for the book and in the mean time there is only so much editing you can do.
The course is structured with readings and videos taken from online sources, with accompanying questions. There does not seem to be any tests or quizzes, but perhaps that is yet to come. What it does have, which is new to me, is an AI Tutor who you can pose questions to. The tutor sits in a window beside the main reading (or viewing videos) window. There is also a left pane with navigation. The AI tutor seems to be powered by a fairly high end system, perhaps a custom installation of Claude or ChatGPT. (Later I learned that it is Claude Haiku with some special training - basically all AI Safety literature).