Do We Have Any Idea What the Risks of Artificial Superintelligence Are?

William Leiss (28 September 2026)

Good Risk Management seeks to anticipate serious harms and control them, not wait for the bad thing to happen and then try to clean up the resultant mess.

Thus, two questions: (1) Do not those who are, in some meaningful sense, working towards the possibility of creating ASI have a moral responsibility to estimate its risks in advance? (2) How can one reasonably anticipate the risks of a technology (ASI) that has no precedent in human history?

(In what follows, I avoid the expectation of a human extinction risk, for example, Yudkowsky & Soares. In order to keep the question open, I assume the ASI might be benign rather than belligerent towards humans, and seek to avoid their extinction, as we now do with reference to the great apes.)

Nevertheless, is it reasonable to assume that a technology based on an advanced degree of intelligence — “smarter than us,” to use the title of Geoffrey Hinton’s forthcoming book — will act in the world in ways not unlike its predecessor (that is, humans, with our brains far superior to those of all other animals)?

If this is a reasonable assumption, the answer is: That new entity (ASI) will, like humans, proceed toward controlling as much of its environment as it can, to provide maximum access to the resources it needs to operate without hindrance up to some limit. (My first book, published in 1972, is The Domination of Nature.) In the case of its predecessor, the result to date has been that humans now dominate the biomass of the earth. The following quotation is from my essay on “Eco-Dominion,” which is the preface to the 2023 edition of my 1972 book:

“The most recent scientific estimates about the magnitude of the accumulated human impacts on the biosphere, expressed in terms of biomass, are: (1) of all mammals now on earth, 60 per cent are livestock, 36 per cent are humans, and 4 per cent are wild; (2) chickens and other poultry are 70 per cent of all birds, the remaining 30 per cent are wild; and (3) since the beginning of human civilization, 83 per cent of wild land mammals and 80 per cent of marine mammals have disappeared. The data is from the most recent scientific calculation in a 2018 journal article published in the Proceedings of the National Academy of Sciences; the interested reader should consult figure 1 in Bar-On et al., “The Biomass Distribution on Earth,” which summarizes the key data in a dramatic graphical format.”[1].

Can we then adapt the human example to the case of ASI? Obviously, with a machine form of intelligence (MI), biomass is in a sense irrelevant. Its crucial resource needs lie in the areas of electricity generation, cooling, machine technology and mineral resources for its computer components, and interlinked global communications.

Are we able to estimate the potential risks of ASI to humans using this conception of its basic needs? And to do so in terms of the formal categories of probability and consequences?

Are we really prepared to advance towards ASI without having a very clear idea of what risks await us there? If so, is this not very likely an exercise in collective suicide?

There is another aspect to consider: the mental and emotional health of humans as they interact with artificial superintelligence. Marcello Guarini has tackled this question in a new paper, “When Machines Make Humans Feel: Emotion, Intelligence, and the Ethics of AI,” for the Journal of Superintelligence (Guarini 2026).

This paper examines the relationship between emotions, meta-emotions, and intelligence. It argues for greater care in the development and deployment of artificial intelligence (AI), especially artificial superintelligence (ASI). The impacts on human beings of both conscious and non-conscious forms of ASI are examined. The potential harm to ASI is considered, but the focus is on harm to humans. An under-discussed alignment consideration for AI is the potential emotional and serious mental health consequences of its misuse. For example, the role AI may play in AI psychosis or other mental health challenges needs greater examination. A case for caution is made in the development of ASI as we are still figuring out the kinds of psychological effects existing AI systems can have on humans, effects that could be (if we are not careful) more harmful in superintelligent systems.

Are we prepared for, or even considered the impact on our own emotional and mental health?

[1] Matthew G. Burgess and Steven D. Gaines, “The Scale of Life and Its Lessons for Humanity,” Proceedings of the National Academy of Sciences 115, no. 25 (2018): 6328–30, https://doi.org/10.1073/pnas.1807019115; citing Yinon M. Bar-On et al., “The Biomass Distribution on Earth,” Proceedings of the National Academy of Sciences 115, no. 25 (2018): 6506–11 and “Supplementary Information Appendix,” https://doi.org/10.1073/pnas.1711842115.

Guarini, Marcello. 2026. “When Machines Make Humans Feel: Emotion, Intelligence, and the Ethics of AI.” Journal of Superintelligence 1 (2): 6. https://doi.org/10.3390/superintelligence1020006.