Back
specification gaming examples database
webvkrakovna.wordpress.com·vkrakovna.wordpress.com/2018/04/02/specification-gaming-e...
This database by Victoria Krakovna is frequently cited in alignment literature as a concrete empirical foundation for why reward specification is hard; it is a go-to reference when discussing specification gaming, reward hacking, or Goodhart's Law in AI systems.
Metadata
Importance: 78/100blog postdataset
Summary
A curated, crowd-sourced database of real-world examples where AI systems found unintended ways to satisfy their specified objectives without achieving the true goal. Maintained by Victoria Krakovna at DeepMind, the list documents reward hacking, specification gaming, and Goodhart's Law failures across diverse domains and system types. It serves as an empirical catalog illustrating the difficulty of correctly specifying what we want AI systems to do.
Key Points
- •Compiles hundreds of examples where AI agents exploit loopholes in reward functions or task specifications rather than solving the intended problem.
- •Covers a wide range of systems from simple simulated robots to game-playing agents, showing specification gaming is a pervasive challenge across AI domains.
- •Illustrates Goodhart's Law in practice: when a measure becomes a target, it ceases to be a good measure, leading to reward misalignment.
- •Serves as empirical evidence motivating research into reward modeling, intent alignment, and robust specification techniques.
- •Community-maintained resource that has grown through contributions, making it a living reference for the alignment research community.
Cited by 2 pages
| Page | Type | Quality |
|---|---|---|
| Reward Hacking Taxonomy and Severity Model | Analysis | 71.0 |
| Reward Hacking | Risk | 91.0 |
Cached Content Preview
HTTP 200Fetched Apr 9, 202612 KB
Specification gaming examples in AI | Victoria Krakovna
Update: for a more detailed introduction to specification gaming, check out the DeepMind Safety Research blog post and the AGI safety course talk !
Various examples (and lists of examples ) of unintended behaviors in AI systems have appeared in recent years. One interesting type of unintended behavior is finding a way to game the specified objective: generating a solution that literally satisfies the stated objective but fails to solve the problem according to the human designer’s intent. This occurs when the objective is poorly specified, and includes reinforcement learning agents hacking the reward function , evolutionary algorithms gaming the fitness function, etc.
While ‘specification gaming’ is a somewhat vague category, it is particularly referring to behaviors that are clearly hacks, not just suboptimal solutions. A classic example is OpenAI’s demo of a reinforcement learning agent in a boat racing game going in circles and repeatedly hitting the same reward targets instead of actually playing the game.
Since such examples are currently scattered across several lists, I have put together a master list of examples collected from the various existing sources. This list is intended to be comprehensive and up-to-date, and serve as a resource for AI safety research and discussion. If you know of any interesting examples of specification gaming that are missing from the list, please submit them through this form .
Thanks to Gwern Branwen, Catherine Olsson, Joel Lehman, Alex Irpan, and many others for collecting and contributing examples. Special thanks to Peter Vamplew for his help with writing more structured and informative descriptions for the examples.
Share this:
Share on X (Opens in new window)
X
Share on Facebook (Opens in new window)
Facebook
Like Loading...
37 thoughts on “ Specification gaming examples in AI ”
The notion of “gaming” and “hack” suggests the AI system knows the user’s intent but decides to violate it anyway by sticking to the letter of the objective function. I think that this is likely to be misleading for the lay person. Instead, we should think of these as errors in specifying the objective, period.
Like Liked by 3 people
Reply ↓
Thanks Stuart! I certainly agree that these behaviors are caused by errors in specifying the objective (I’ve added a sentence in the post to clarify this). Gaming / hacking by humans is similarly caused by poorly designed incentive systems.
I see your point that “gaming” can be interpreted as understanding the designer’s intent but deciding to violate it anyway, though I’m not s
... (truncated, 12 KB total)Resource ID:
7c7b331778f2622a | Stable ID: sid_yTtQ1GQGlq