Back
My views on “doom”
webAuthor
paulfchristiano
Credibility Rating
3/5
Good(3)Good quality. Reputable source with community review or editorial standards, but less rigorous than peer-reviewed venues.
Rating inherited from publication venue: LessWrong
This post by Paul Christiano (founder of Alignment Research Center) is frequently cited in AI safety discussions as a concrete quantitative risk estimate from a prominent insider, offering a useful baseline for comparing risk assessments across the field.
Forum Post Details
Karma
252
Comments
38
Forum
lesswrong
Forum Tags
Existential riskForecasts (Specific Predictions)AI
Metadata
Importance: 72/100blog postprimary source
Summary
Paul Christiano shares his personal probabilistic estimates for AI-related catastrophic outcomes, including a 22% overall probability of AI takeover. He carefully distinguishes between extinction risk, existential risk without extinction, and loss of human control over civilization's future, while emphasizing these are rough personal estimates subject to significant uncertainty.
Key Points
- •Estimates 22% probability of AI takeover overall: 15% from human-built AI systems and 7% from other sources.
- •Distinguishes carefully between extinction, existential risk (bad futures without extinction), and loss of human control over civilization's trajectory.
- •Clarifies the difference between dying from AI directly versus dying from other causes that AI accelerates.
- •Emphasizes estimates are rough personal views, not calibrated predictions, and fluctuate as new considerations arise.
- •Provides a rare public quantitative breakdown of AI risk from a leading alignment researcher, useful as a reference point.
Cited by 1 page
| Page | Type | Quality |
|---|---|---|
| Why Alignment Might Be Hard | Argument | 69.0 |
Cached Content Preview
HTTP 200Fetched Apr 10, 20264 KB
# My views on “doom”
By paulfchristiano
Published: 2023-04-27
I’m often asked: “what’s the probability of a really bad outcome from AI?”
There are many different versions of that question with different answers. In this post I’ll try to answer a bunch of versions of this question all in one place.
#### Two distinctions
Two distinctions often lead to confusion about what I believe:
* One distinction is between **dying** (“_extinction_ risk”) and **having a bad future** (“_existential_ risk”). I think there’s a good chance of bad futures without extinction, e.g. that AI systems take over but don’t kill everyone.
* An important subcategory of “bad future” is “AI takeover:” an outcome where the world is governed by AI systems, and we weren’t able to build AI systems who share our values or care a lot about helping us. This need not result in humans dying, and it may not even be an objectively terrible future. But it does mean that humanity gave up control over its destiny, and I think in expectation it’s pretty bad.
* A second distinction is between **dying now** and **dying later.** I think that there’s a good chance that we don’t die from AI, but that AI and other technologies greatly accelerate the rate of change in the world and so something else kills us shortly later. I wouldn’t call this “from AI” but I do think it happens soon in calendar time and I’m not sure the distinction is comforting to most people.
#### Other caveats
I’ll give my beliefs in terms of probabilities, but these really are just best guesses — the point of numbers is to quantify and communicate what I believe, not to claim I have some kind of calibrated model that spits out these numbers.
Only one of these guesses is even really related to my day job (the 15% probability that AI systems built by humans will take over). For the other questions I’m just a person who’s thought about it a bit in passing. I wouldn’t recommend deferring to the 15%, but _definitely_ wouldn’t recommend deferring to anything else.
A final source of confusion is that I give different numbers on different days. Sometimes that’s because I’ve considered new evidence, but normally it’s just because these numbers are just an imprecise quantification of my belief that changes from day to day. One day I might say 50%, the next I might say 66%, the next I might say 33%.
I’m giving percentages but you should treat these numbers as having 0.5 significant figures.
#### My best guesses
Probability of an AI takeover: **22%**
* Probability that humans build AI systems that take over: **15%
**(Including anything that happens before human cognitive labor is basically obsolete.)
* Probability that the AI we build doesn’t take over, but that _it_ builds even smarter AI and there is a takeover some day further down the line: **7%**
Probability that most humans die within 10 years of building powerful AI (powerful enough to make human labor obsolete): **20%**
* Probability that most humans
... (truncated, 4 KB total)Resource ID:
ed73cbbe5dec0db9 | Stable ID: sid_6pl1H2I0cQ