Response to Aschenbrenner's "Situational Awareness"

web

2024·EA Forum·forum.effectivealtruism.org/posts/RTHFCRLv34cewwMr6/respo...

Author

RobBensinger

Credibility Rating

3/5

Good(3)

Good quality. Reputable source with community review or editorial standards, but less rigorous than peer-reviewed venues.

Rating inherited from publication venue: EA Forum

This EA Forum post responds to Leopold Aschenbrenner's widely-read 'Situational Awareness' document (2024), which argued AGI is imminent and framed it as a national security priority; this response represents the AI safety community's critical engagement with that thesis.

Metadata

Importance: 55/100commentary

Summary

A critical response published on the EA Forum to Leopold Aschenbrenner's influential 'Situational Awareness' document, which argued for rapid AGI development with a national security framing. The response likely challenges key assumptions about AI timelines, safety, geopolitical strategy, or the framing of AGI as primarily a US-China competition.

Key Points

•Engages critically with Aschenbrenner's 'Situational Awareness' essay, which predicted rapid AGI development and emphasized US national security imperatives.
•Likely challenges the framing of AGI development as primarily a geopolitical race requiring speed over safety.
•Addresses concerns about how the 'race dynamics' narrative could undermine international coordination and AI safety efforts.
•Offers an EA/AI safety community perspective on the risks of Aschenbrenner's policy recommendations.
•Contributes to ongoing debate about whether accelerating AI development for strategic advantage is wise given existential risk considerations.

Cached Content Preview

HTTP 200Fetched Apr 10, 20266 KB

# Response to Aschenbrenner's "Situational Awareness"
By RobBensinger
Published: 2024-06-06
([*Cross-posted from Twitter.*](https://x.com/robbensinger/status/1798845199382429697))

My take on Leopold Aschenbrenner's new [report](https://situational-awareness.ai/): I think Leopold gets it right on a bunch of important counts.

Three that I especially care about:

1.  Full AGI and ASI [soon](https://www.lesswrong.com/posts/cCMihiwtZx7kdcKgt/comments-on-carlsmith-s-is-power-seeking-ai-an-existential#Timelines). (I think his [arguments](https://situational-awareness.ai/from-gpt-4-to-agi/) for this have a lot of [holes](https://twitter.com/ESYudkowsky/status/1798105252375503176), but he gets the basic point that superintelligence looks 5 or 15 years off rather than 50+.)
2.  This technology is an overwhelmingly huge deal, and if we play our cards wrong we're all dead.
3.  Current developers are indeed fundamentally unserious about the core risks, and need to make IP security and closure a top priority.

I especially appreciate that the report seems to *get it* when it comes to our basic strategic situation: it gets that we may only be a few years away from a truly world-threatening technology, and it speaks very candidly about the implications of this, rather than soft-pedaling it to the degree that public writings on this topic almost always do. I think that's a valuable contribution all on its own.

Crucially, however, I think Leopold gets the wrong answer on the question "is alignment tractable?". That is: OK, we're on track to build vastly smarter-than-human AI systems in the next decade or two. How realistic is it to think that we can control such systems?

Leopold acknowledges that we currently only have guesswork and half-baked ideas on the technical side, that this field is extremely young, that many aspects of the problem look impossibly difficult (see attached image), and that there's a strong chance of this research operation getting us all killed. "To be clear, given the stakes, I think 'muddling through' is in some sense a terrible plan. But it might be all we’ve got." *Controllable* superintelligent AI is a far more speculative idea at this point than superintelligent AI itself.

![Image](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/Yig9oa4zGE97xM2os/fr5zhtc02vrbyhwibcay)

I think this report is drastically mischaracterizing the situation. ‘This is an awesome exciting technology, let's race to build it so we can reap the benefits and triumph over our enemies’ is an appealing narrative, but it requires the facts on the ground to shake out very differently than how the field's trajectory currently looks.

The more *normal* outcome, if the field continues as it has been, is: if anyone builds it, everyone dies.

This is not a national security issue of the form ‘exciting new tech that can give a country an economic or military advantage’; it's a national security issue of the form ‘we've found a way t

... (truncated, 6 KB total)

Resource ID: c575273aae0df6b7 | Stable ID: sid_furbJf47JZ