Another petition

AI Safety

Yesterday we learned that almost 1200 employees of the “Frontier Labs” (code for Google DeepMind, Anthropic, OpenAI, and Facebook Meta) have signed a new petition. The petition, called “Pacing the Frontier,” is not a call to stop development but it does seek to ensure there is an option to pace (slow down? Stop, perhaps?) development if necessary. Here’s the call:

We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”

Will this matter? According to The Next Web:

The names are the story. Signatories include Anthropic chief executive Dario Amodei and its co-founders Jared Kaplan and Jack Clark. So did OpenAI chief scientist Jakub Pachocki, Meta chief scientist Shengjia Zhao, and Google’s head of AI safety, Anca Dragan. Both Anthropic and OpenAI endorsed the letter officially.

What spurred this initiative? It looks like the trigger was recursive self-improvement, something we have focused on our book as particularly dangerous and a likely prelude to loss of control (see Leiss and Smith, From Tool to Actor (forthcoming), Chapter 4: The Superhuman Machine). Still, it comes at a bit of an awkward time, as the major labs have also been arguing for openness as the solution to AI safety. It also comes on the heels of a major “escape” out of a Frontier lab and hack into another AI company (see my “crash test” post).

Will this matter? We have had petitions before. I think I’ve signed at least three of them. According to The Next Web, the difference here is that this is not an open petition. These are the insiders.

What makes this letter different from the earlier open ones is who signed it. Not outside critics, but the engineers and executives closest to the frontier. One OpenAI safety researcher, Leo Gao, put the stakes bluntly: “the world is locked in a deadly race towards an intelligence explosion.” His fix is the one the whole letter turns on. “To survive, we must coordinate to slow down the race.”

I don’t really buy that, as I’ve seen Hinton, Bengio, et al. on the previous petitions. How are these names more important? Why is ‘pacing’ more palatable than ‘pause’ (or stop, for that matter)?

Commentary

  1. Zvi Mowshowitz, our go to guy on matters such as these, is more impressed that I am about the letter.

  2. Nate Soares, ever a skeptic, remains skeptical. (That’s a chain of tweets, so a bit awkward to follow - the best option is to actually read the annotated/condensed version inside Zvi’s post, above.) To get a sense of Soares’ take on things:

    “To realize AI’s potential…” (part of the petition’s preamble.)

    Bullshit. AI has the potential to wipe us out (and in fact, that negative potential is much easier to realize than its potential to, e.g., cure aging for us). Opening with “To realize AI’s potential” is a framing device that bakes in the assumption that this is a grand good thing that just needs a little bit of extra care, rather than a risky difficult-to-manage explosive that would wipe us off the planet if treated with anything short of extreme caution.