Risks from Learned Optimization

[1906.01820] Risks from Learned Optimization in Advanced Machine Learning Systems 
 
 
 
 
 
 
 
 
 
 
 

 
 
 

 
 
 
 
 
 
 Risks from Learned Optimization 
 in Advanced Machine Learning Systems

 
 
 Evan Hubinger
 Alphabetical order. Equal contribution.
 
 
 
 Chris van Merwijk 1 1 footnotemark: 1 
 
 
 
 
 Vladimir Mikulik 1 1 footnotemark: 1 
 
 
 
 
 Joar Skalse 1 1 footnotemark: 1 
 
 
 
 
 and Scott Garrabrant
 With special thanks to Paul Christiano, Eric Drexler, Rob Bensinger, Jan Leike, Rohin Shah, William Saunders, Buck Shlegeris, David Dalrymple, Abram Demski, Stuart Armstrong, Linda Linsefors, Carl Shulman, Toby Ord, Kate Woolverton, and everyone else who provided feedback on earlier versions of this paper.
 
 
 
 (June 11, 2019) 

 
 Abstract

 We analyze the type of learned optimization that occurs when a learned model (such as a neural network) is itself an optimizer—a situation we refer to as mesa-optimization, a neologism we introduce in this paper. We believe that the possibility of mesa-optimization raises two important questions for the safety and transparency of advanced machine learning systems. First, under what circumstances will learned models be optimizers, including when they should not be? Second, when a learned model is an optimizer, what will its objective be—how will it differ from the loss function it was trained under—and how can it be aligned? In this paper, we provide an in-depth analysis of these two primary questions and provide an overview of topics for future research.

 

 
 
 1 Introduction

 
 \cftchapterprecistoc 
 We introduce the concept of mesa-optimization as well as many relevant terms and concepts related to it, such as what makes a system an optimizer, the difference between base and mesa- optimizers, and our two key problems: unintended optimization and inner alignment.

 
 
 In machine learning, we do not manually program each individual parameter of our models. Instead, we specify an objective function that captures what we want the system to do and a learning algorithm to optimize the system for that objective. In this paper, we present a framework that distinguishes what a system is optimized to do (its “purpose”), from what it optimizes for (its “goal”), if it optimizes for anything at all. While all AI systems are optimized for something (have a purpose), whether they actually optimize for anything (pursue a goal) is non-trivial. We will say that a system is an optimizer if it is internally searching through a search space (consisting of possible outputs, policies, plans, strategies, or similar) looking for those elements that score high according to some objective function that is explicitly represented within the system. Learning algorithms in machine learning are optimizers because they search through a space of possible parameters—e.g. neural network weights—and improve the parameters with respect to some objective. Planning algorithms are also optimizers, since they search through p

... (truncated, 98 KB total)