Superintelligence Cannot be Contained: Lessons from Computability Theory

Superintelligence Cannot be Contained: Lessons from Computability Theory. Manuel Alfonseca et al. Journal of Artificial Intelligence Research, vol 70, Jan 5, 2021. https://doi.org/10.1613/jair.1.12202

Abstract: Superintelligence is a hypothetical agent that possesses intelligence far surpassing that of the brightest and most gifted human minds. In light of recent advances in machine intelligence, a number of scientists, philosophers and technologists have revived the discussion about the potentially catastrophic risks entailed by such an entity. In this article, we trace the origins and development of the neo-fear of superintelligence, and some of the major proposals for its containment. We argue that total containment is, in principle, impossible, due to fundamental limits inherent to computing itself. Assuming that a superintelligence will contain a program that includes all the programs that can be executed by a universal Turing machine on input potentially as complex as the state of the world, strict containment requires simulations of such a program, something theoretically (and practically) impossible.

More terrible than either of these tales is the fable of the monkey’s paw (Jacobs, 1902), written by W. W. Jacobs, an English writer of the beginning of the [20th] century. A retired English working-man is sitting at his table with his wife and a friend, a returned British sergeant-major from India. The sergeant-major shows his hosts an amulet in the form of a dried, wizened monkey’s paw... [which has] the power of granting three wishes to each of three people... The last [wish of the first owner] was for death... His friend... wishes to test its powers. His first [wish] is for 200 pounds. Shortly thereafter there is a knock at the door, and an official of the company by which his son is employed enters the room. The father learns that his son has been killed in the machinery, but that the company... wishes to pay the father the sum of 200 pounds... The grief-stricken father makes his second wish -that his son may return- and when there is another knock at the door... something appears... the ghost of the son. The final wish is that the ghost should go away. In these stories, the point is that the agencies of magic are literal-minded... The new agencies of the learning machine are also literal-minded. If we program a machine... and ask for victory and do not know what we mean by it, we shall find the ghost knocking at our door.