Specification Gaming: The Flip Side of AI Ingenuity

[Skip to main content](https://deepmind.google/blog/specification-gaming-the-flip-side-of-ai-ingenuity/#page-content)

April 21, 2020
Research

# Specification gaming: the flip side of AI ingenuity

Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, Shane Legg

Share

![Two golden apples sitting on a pale green surface.](https://lh3.googleusercontent.com/LaMG-UHc_7mHJiknImfYpSxxmIocmr3vBDz2iKC_pptg5RpsLCtxgHkk0oMxp_-dTBAmiSnJQveFD0V83c8ud-yLxM4DQNSvRMhn_3oU22I=w1440-h810-n-nu)

**Specification gaming** is a behaviour that satisfies the literal specification of an objective without achieving the intended outcome. We have all had experiences with specification gaming, even if not by this name. Readers may have heard the myth of [King Midas](https://en.wikipedia.org/wiki/Midas) and the golden touch, in which the king asks that anything he touches be turned to gold - but soon finds that even food and drink turn to metal in his hands. In the real world, when rewarded for doing well on a homework assignment, a student might copy another student to get the right answers, rather than learning the material - and thus exploit a loophole in the task specification.

This problem also arises in the design of artificial agents. For example, a reinforcement learning agent can find a shortcut to getting lots of reward without completing the task as intended by the human designer. These behaviours are common, and we have [collected](http://tinyurl.com/specification-gaming) around 60 examples so far (aggregating [existing](https://arxiv.org/abs/1803.03453) [lists](https://www.gwern.net/Tanks#alternative-examples) and ongoing [contributions](https://docs.google.com/forms/d/e/1FAIpQLSeQEguZg4JfvpTywgZa3j-1J-4urrnjBVeoAO7JHIH53nrBTA/viewform) from the AI community). In this post, we review possible causes for specification gaming, share examples of where this happens in practice, and argue for further work on principled approaches to overcoming specification problems.

Let's look at an example. In a [Lego stacking task](https://arxiv.org/abs/1704.03073), the desired outcome was for a red block to end up on top of a blue block. The agent was rewarded for the height of the bottom face of the red block when it is not touching the block. Instead of performing the relatively difficult maneuver of picking up the red block and placing it on top of the blue one, the agent simply flipped over the red block to collect the reward. This behaviour achieved the stated objective (high bottom face of the red block) at the expense of what the designer actually cares about (stacking it on top of the blue one).

![Animation of a robotic arm with one blue and one red lego piece. It flips over the red lego piece.](https://lh3.googleusercontent.com/Fruz7UaMO-nOZ1H3rKyGoTI8PFZqPdk16kkVgb5eIL_MZyMVI5TcfJny1_WZNuIMW2qIPP7cVWeEzaVbSSYFFnBKhYTjyQtHcT9qYPNg=w1440)

Source: Data-Efficient Deep Reinforcement Learning for Dexterou

... (truncated, 14 KB total)