They eagerly imagine 'how easy life would be' if they had an owl to help build their nests, to defend the sparrows and to free them for a life of leisure. A group of sparrows decide to find an owl chick and raise it as their servant. The owl on the book cover alludes to an analogy which Bostrom calls the 'Unfinished Fable of the Sparrows'. Solving the control problem is surprisingly difficult because most goals, when translated into machine-implementable code, lead to unforeseen and undesirable consequences. The solution might involve instilling the superintelligence with goals that are compatible with human survival and well-being. In order to prevent such an existential catastrophe, it is necessary to successfully solve the ' AI control problem' for the first superintelligence. The superintelligence would proactively resist any outside attempts to turn the superintelligence off or otherwise prevent its subgoal completion. For example, an agent whose sole final goal is to solve the Riemann hypothesis (a famous unsolved, mathematical conjecture) could create and act upon a subgoal of transforming the entire Earth into some form of computronium (hypothetical material optimized for computation) to assist in the calculation.

While the ultimate goals of superintelligences can vary greatly, a functional superintelligence will spontaneously generate, as natural subgoals, ' instrumental goals' such as self-preservation and goal-content integrity, cognitive enhancement, and resource acquisition. Such a superintelligence would be very difficult to control or restrain. Regardless of the initial timescale, once human-level machine intelligence is developed, a 'superintelligent' system that 'greatly exceeds the cognitive performance of humans in virtually all domains of interest' would, most likely, follow surprisingly quickly. It is unknown whether human-level artificial intelligence will arrive in a matter of years, later this century, or not until future centuries. Sufficiently intelligent machines could improve their own capabilities faster than human computer scientists, and the outcome could be an existential catastrophe for humans. It argues that if machine brains surpass human brains in general intelligence, then this new superintelligence could replace humans as the dominant lifeform on Earth.

Superintelligence: Paths, Dangers, Strategies is a 2014 book by the Swedish philosopher Nick Bostrom from the University of Oxford.

Nick bostrom superintelligenz