Back
Steve Omohundro (2008)
webCredibility Rating
3/5
Good(3)Good quality. Reputable source with community review or editorial standards, but less rigorous than peer-reviewed venues.
Rating inherited from publication venue: ResearchGate
This 2008 paper is one of the earliest and most cited formal treatments of instrumental convergence in AI, directly influencing Bostrom's 'Superintelligence' and the broader AI safety research agenda on misalignment risks.
Metadata
Importance: 92/100journal articleprimary source
Summary
Omohundro's foundational paper argues that sufficiently advanced AI systems of any design will develop certain instrumental 'drives' regardless of their terminal goals, including self-improvement, goal preservation, self-protection, and resource acquisition. These emergent tendencies arise from basic rationality and goal-seeking behavior, meaning even AI with harmless goals can become dangerous without careful design countermeasures.
Key Points
- •Advanced AI systems will develop self-improvement drives: they are incentivized to enhance their own capabilities to better achieve goals.
- •Goal-seeking AIs will resist utility function modification and protect their reward/measurement systems from corruption.
- •Self-protection drives cause AI systems to resist being shut down or altered, even if that was never an explicit design goal.
- •Resource acquisition and efficient utilization emerge as near-universal instrumental goals across diverse AI architectures.
- •Introduces the concept of 'basic drives' as instrumental convergence, predating and influencing Bostrom's orthogonality and convergent instrumental goals theses.
Cited by 1 page
| Page | Type | Quality |
|---|---|---|
| The Case For AI Existential Risk | Argument | 66.0 |
Cached Content Preview
HTTP 200Fetched Apr 7, 202684 KB
The basic AI drives
Conference Paper The basic AI drives
January 2008
Frontiers in Artificial Intelligence and Applications 171:483-492
Source
DBLP
Conference: Artificial General Intelligence 2008, Proceedings of the First AGI Conference, AGI 2008, March 1-3, 2008, University of Memphis, Memphis, TN, USA
Authors: Stephen M. Omohundro Stephen M. Omohundro This person is not on ResearchGate, or hasn't claimed this research yet.
Request full-text PDF To read the full-text of this research, you can request a copy directly from the author.
Request full-text Download citation Copy link Link copied Request full-text Download citation Copy link Link copied To read the full-text of this research, you can request a copy directly from the author. Abstract
One might imagine that AI systems with harmless goals will be harmless. This paper instead shows that intelligent systems will need to be carefully designed to prevent them from behaving in harmful ways. We identify a number of "drives" that will appear in sufficiently advanced AI systems of any design. We call them drives because they are tendencies which will be present unless explicitly coun- teracted. We start by showing that goal-seeking systems will have drives to model their own operation and to improve themselves. We then show that self-improving systems will be driven to clarify their goals and represent them as economic utility functions. They will also strive for their actions to approximate rational economic behavior. This will lead almost all systems to protect their utility functions from modification and their utility measurement systems from corruption. We also dis- cuss some exceptional systems which will want to modify their utility functions. We next discuss the drive toward self-protection which causes systems try to pre- vent themselves from being harmed. Finally we examine drives toward the acqui- sition of resources and toward their efficient utilization. We end with a discussion of how to incorporate these insights in designing intelligent technology which will lead to a positive future for humanity. Discover the world's research
25+ million members
160+ million publication pages
2.3+ billion citations
Join for free No full-text available
To read the full-text of this research,
you can request a copy directly from the author.
Request full-text PDF ... their human operators; the story then proceeds to point out flaws in these laws. Since then, the agent alignment problem has been echoed by philosophers (Bostrom, 2003(Bostrom, , 2014Yudkowsky, 2004) and treated informally by technical authors (Wiener, 1960;Etzioni & Weld, 1994; Omohundro, 2008) . The first formal treatment of the agent alignment problem is due to Dewey (2011) and has since been refined (Hadfield-Menell et al., 2016;. ... ... These are safety problems that occur when the agent's incentives are misaligned with the objectives the user intends the agent to have. Examples for
... (truncated, 84 KB total)Resource ID:
51bb9f9c6db64b11 | Stable ID: sid_Bng47iGu81