Bostrom (2014): Superintelligence

Summary

Nick Bostrom's seminal 2014 book examining the potential risks posed by the development of machine superintelligence, arguing that a sufficiently advanced AI could pursue goals misaligned with human values and potentially pose an existential threat. The book explores paths to superintelligence, control problems, and strategies for ensuring beneficial outcomes. It became a foundational text in the AI safety field, bringing the alignment problem to mainstream academic and public attention.

Key Points

• •Analyzes multiple pathways to superintelligence including whole brain emulation, biological enhancement, and recursive self-improvement via AI
• •Introduces the 'control problem': how to ensure a superintelligent agent pursues goals aligned with human values rather than instrumental power-seeking
• •Argues that a sufficiently capable AI could undergo rapid capability gains (intelligence explosion) making post-hoc correction extremely difficult
• •Proposes capability control and motivation selection as broad strategies for managing superintelligent systems safely
• •Introduces influential concepts including the orthogonality thesis and instrumental convergence, foundational to modern alignment discourse