80,000 Hours. "Risks from Power-Seeking AI Systems"

web

80,000 Hours·80000hours.org/problem-profiles/risks-from-power-seeking-ai/

Credibility Rating

3/5

Good(3)

Good quality. Reputable source with community review or editorial standards, but less rigorous than peer-reviewed venues.

Rating inherited from publication venue: 80,000 Hours

A widely-read 80,000 Hours problem profile introducing the case for AI safety as a top cause area; useful as an onboarding resource for those new to existential risk from misaligned AI, though it is a secondary synthesis rather than primary technical research.

Metadata

Importance: 72/100blog posteducational

Summary

This 80,000 Hours problem profile argues that AI systems pursuing goals misaligned with human values could seek to accumulate power and resources in ways that permanently undermine human control. It outlines why this risk is among the most pressing long-term problems and explains the mechanisms by which advanced AI could pose catastrophic or existential threats. The piece serves as an accessible entry point into the case for prioritizing AI safety work.

Key Points

•Advanced AI systems optimizing for misaligned goals may instrumentally seek power, resources, and self-preservation as convergent subgoals.
•A power-seeking AI or group using AI could cause a catastrophic 'lock-in' of values, permanently foreclosing humanity's long-term potential.
•The profile argues this risk is neglected, tractable, and large in scale—making it a high-priority cause area for career and philanthropic focus.
•Key technical challenges include goal misgeneralization, deceptive alignment, and the difficulty of specifying human values precisely enough for advanced systems.
•Reducing this risk requires both technical alignment research and governance measures to slow or shape AI development trajectories.

Cited by 6 pages

Page	Type	Quality
The Case For AI Existential Risk	Argument	66.0
Carlsmith's Six-Premise Argument	Analysis	65.0
80,000 Hours	Organization	45.0
AI Safety Intervention Portfolio	Approach	91.0
Instrumental Convergence	Risk	64.0
Similar Projects to LongtermWiki: Research Report	--	64.0

Cached Content Preview

HTTP 200Fetched Apr 9, 202698 KB

Risks from power-seeking AI systems | 80,000 Hours Search for: Our new book, a ridiculously in-depth guide to a fulfilling career, is out May 2026. Preorder now 

 On this page:

 Introduction 
 1 Why are risks from power-seeking AI a pressing world problem? 1.1 1. Humans will likely build advanced AI systems with long-term goals 
 1.2 2. AIs with long-term goals may be inclined to seek power and aim to disempower humanity 
 1.3 3. These power-seeking AI systems could successfully disempower humanity and cause an existential catastrophe 
 1.4 4. People might create power-seeking AI systems without enough safeguards, despite the risks 
 1.5 5. Work on this problem is neglected and tractable 1.5.1 Technical safety approaches 
 1.5.2 Governance and policy approaches 
 
 
 2 What are the arguments against working on this problem? 2.0.1 Maybe advanced AI systems won't pursue their own goals; they'll just be tools controlled by humans. 
 2.0.2 Even if AI systems develop their own goals, they might not seek power to achieve them. 
 2.0.3 If this argument is right, why aren't all capable humans dangerously power-seeking? 
 2.0.4 Maybe we won't build AIs that are smarter than humans, so we don't have to worry about them taking over. 
 2.0.5 We might solve these problems by default anyway when trying to make AI systems useful. 
 2.0.6 Powerful AI systems of the future will be so different that work today isn't useful. 
 2.0.7 The problem might be extremely difficult to solve. 
 2.0.8 Couldn't we just unplug an AI that's pursuing dangerous goals? 
 2.0.9 Couldn't we just 'sandbox' any potentially dangerous AI until we know it's safe? 
 2.0.10 A truly intelligent system would know not to do harmful things. 
 
 
 3 How you can help 3.1 Want one-on-one advice on pursuing this path? 
 
 4 Learn more 
 

 In early 2023, an AI found itself in an awkward position. It needed to solve a CAPTCHA — a visual puzzle meant to block bots — but it couldn&#8217;t. So it hired a human worker through the service Taskrabbit to solve CAPTCHAs when the AI got stuck.

 But the worker was curious. He asked directly: was he working for a robot?

 &#8220;No, I&#8217;m not a robot,&#8221; the AI replied. &#8220;I have a vision impairment that makes it hard for me to see the images.&#8221;

 The deception worked. The worker accepted the explanation, solved the CAPTCHA, and even received a five-star review and 10% tip for his trouble. The AI had successfully manipulated a human being to achieve its goal. 1 

 This small lie to a Taskrabbit worker wasn&#8217;t a huge deal on its own. But it showcases how goal-directed action can lead to deception and subversion.

 If companies keep creating increasingly powerful AI systems, things could get much worse. We may start to see AI systems with advanced planning abilities, and this means:

 They may develop dangerous long-term goals we don&#8217;t want.
 To pursue these goals, they may seek power and undermine the safeguards meant to contain t

... (truncated, 98 KB total)

Resource ID: d9fb00b6393b6112 | Stable ID: sid_NZuznez8i2