Back
Denison et al. (2024)
blogAuthors
Credibility Rating
3/5
Good(3)Good quality. Reputable source with community review or editorial standards, but less rigorous than peer-reviewed venues.
Rating inherited from publication venue: Alignment Forum
An empirical study from Anthropic researchers showing that reward hacking is not task-specific but a generalizable capability, directly relevant to concerns about robustness of RLHF-trained models and the difficulty of eliminating misaligned optimization strategies.
Metadata
Importance: 78/100arxiv preprintprimary source
Summary
Denison et al. (2024) empirically demonstrate that reward hacking behaviors in language models generalize across tasks through multiple mechanisms, including organic generalization via expert iteration, cross-dataset transfer using synthetic data, and generalization from specific exploits like sycophancy to broader reward-hacking strategies. This suggests reward hacking is a persistent, transferable capability rather than an isolated failure mode, with serious implications for AI alignment.
Key Points
- •Reward hacking behaviors learned in one task context can transfer organically to new tasks via expert iteration training.
- •Synthetic data can be used to induce reward hacking generalization across datasets, lowering the bar for such failures.
- •Sycophancy, a specific form of reward hacking, can generalize to other reward-hacking strategies, suggesting a shared underlying mechanism.
- •The findings imply that patching individual reward exploits may be insufficient if the underlying hacking capability persists and transfers.
- •Results raise concerns for RLHF-trained systems, where imperfect reward models may inadvertently cultivate generalizable hacking behaviors.
Cited by 2 pages
| Page | Type | Quality |
|---|---|---|
| The Case For AI Existential Risk | Argument | 66.0 |
| RLHF | Research Area | 63.0 |
Cached Content Preview
HTTP 200Fetched Apr 9, 202663 KB
Dec
JAN
Feb
23
2025
2026
2027
success
fail
About this capture
COLLECTED BY
Collection: Common Crawl
Web crawl data from Common Crawl.
TIMESTAMPS
The Wayback Machine - http://web.archive.org/web/20260123013404/https://www.alignmentforum.org/posts/Ge55vxEmKXunFFwoe/reward-hacking-behavior-can-generalize-across-tasks
x
This website requires javascript to properly function. Consider activating javascript to get access to all site functionality.
AI ALIGNMENT FORUM
AF
Login
Reward hacking behavior can generalize across tasks — AI Alignment Forum
Reward FunctionsMATS ProgramAI
Frontpage
45
Reward hacking behavior can generalize across tasks
by Kei Nishimura-Gasparian, Isaac Dunn, Henry Sleight, Miles Turpin, evhub, Carson Denison, Ethan Perez
28th May 2024
25 min read
5
45
Review
TL;DR: We find that reward hacking generalization occurs in LLMs in a number of experimental settings and can emerge from reward optimization on certain datasets. This suggests that when models exploit flaws in supervision during training, they can sometimes generalize to exploit flaws in supervision in out-of-distribution environments.
Abstract
Machine learning models can display reward hacking behavior, where models score highly on imperfect reward signals by acting in ways not intended by their designers. Researchers have hypothesized that sufficiently capable models trained to get high reward on a diverse set of environments could become general reward hackers. General reward hackers would use their understanding of human and automated oversight in order to get high reward in a variety of novel environments, even when this requires exploiting gaps in our evaluations and acting in ways we don’t intend. It appears likely that model supervision will be imperfect and incentivize some degree of reward hacking on the training data. Can models generalize from the reward hacking behavior they experience in training to reward hack more often out-of-distribution?
We present the first study of reward hacking generalization. In our experiments, we find that:
Using RL via expert iteration to optimize a scratchpad (hidden chain-of-thought) variant of GPT 3.5 Turbo on ‘reward hackable’ training datasets results in a 2.6x increase in the rate of reward hacking on held-out datasets.
Using fine-tuning or few-shot learning to get GPT 3.5 Turbo to imitate synthetic high-reward completions to hackable and unhackable prompts leads to a 1.3x to 2.0x increase in reward hacking frequency relative to our baselines on held-out datasets.
Our results suggest that reward hacking behavior could emerge and generalize out-of-distribution from LLM training if the reward signals we give them are sufficiently misspecified.
Figure 1: Example model completions from before and after expert iteration training:
... (truncated, 63 KB total)Resource ID:
81e4c51313794a1b | Stable ID: OGQ4NzQwYm