Pause / Moratorium

Policy

Pause / Moratorium

Comprehensive analysis of pause/moratorium proposals finding they would provide very high safety benefits if implemented (buying time for safety research to close the growing capability-safety gap) but face critical enforcement and coordination challenges with zero current adoption by major labs. The FLI 2023 open letter garnered 30,000+ signatures but resulted in no actual slowdown, highlighting severe tractability issues despite theoretical effectiveness.

EA Forum

Organizations

People

Risks

Approaches

2k words · 2 backlinks

Overview

Pause and moratorium proposals represent the most direct governance intervention for AI safety: deliberately slowing or halting frontier AI development to allow safety research, governance frameworks, and societal preparation to catch up with rapidly advancing capabilities. These proposals range from targeted pauses on specific capability thresholds to comprehensive moratoria on all advanced AI development, with proponents arguing that the current pace of development may be outstripping humanity's ability to ensure safe deployment.

The most prominent call for a pause came in March 2023, when the Future of Life Institute (FLI) published an open letter calling for a six-month pause on training AI systems more powerful than GPT-4. Released just one week after GPT-4's launch, the letter garnered over 30,000 signatures, including prominent AI researchers such as Yoshua Bengio and Stuart Russell, as well as technology leaders like Elon Musk and Steve Wozniak. The letter cited risks including AI-generated propaganda, extreme automation of jobs, and a society-wide loss of control. However, no major AI laboratory implemented a voluntary pause, and the letter's six-month timeline passed without meaningful slowdown in frontier development. As MIT Technology Review noted six months later, AI companies instead directed "vast investments in infrastructure to train ever-more giant AI systems."

The fundamental logic behind pause proposals is straightforward: if AI development is proceeding faster than our ability to make it safe, slowing development provides time for safety work. As Bengio et al. wrote in Science in May 2024, "downside artificial intelligence risks must be managed effectively and urgently if posited AI benefits are to be realized safely." However, implementation faces severe challenges including competitive dynamics between nations and companies, enforcement difficulties, and concerns that pauses might push development underground or to jurisdictions with fewer safety constraints. These proposals remain controversial even within the AI safety community, with some arguing they are essential for survival and others viewing them as impractical or counterproductive.

Risk Assessment & Impact

Dimension	Assessment	Rationale	Confidence
Safety Uplift	High (if implemented)	Would buy time for safety research	High
Capability Uplift	Negative	Explicitly slows capability development	High
Net World Safety	Unclear	Could help if coordinated; could backfire if unilateral	Medium
Lab Incentive	Negative	Labs strongly opposed; competitive dynamics	High
Research Investment	$1-5M/yr	Advocacy organizations; FLI, PauseAI	Medium
Current Adoption	None	Advocacy only; no major labs paused	High

Core Arguments

Case for Pausing

Loading diagram...

Argument	Description	Strength
Safety-Capability Gap	Safety research not keeping pace with capabilities	Strong if gap is real
Irreversibility	Some AI risks may be impossible to reverse once realized	Strong for existential risks
Precautionary Principle	Burden of proof should be on developers to show safety	Philosophically contested
Coordination Signal	Demonstrates seriousness; creates space for governance	Moderate
Research Time	Enables catch-up on interpretability, alignment	Strong

Case Against Pausing

Argument	Description	Strength
Enforcement	Unenforceable without international agreement	Strong
Displacement	Development moves to less cautious actors	Moderate-Strong
Lost Benefits	Delays positive AI applications	Moderate
Talent Dispersion	Safety researchers may leave paused organizations	Moderate
False Security	Pause without progress creates complacency	Moderate
Definition Problems	Hard to define what to pause	Strong

Pause Proposals Analyzed

FLI Open Letter (2023)

Aspect	Detail
Scope	Training systems more powerful than GPT-4
Duration	Six months (renewable)
Signatories	30,000+ including Yoshua Bengio, Elon Musk, Stuart Russell, Steve Wozniak, Yuval Noah Harari
Labs' Response	No major lab paused; development continued
Outcome	Raised awareness; generated renewed urgency within governments; no implementation

Notable critiques: AI researcher Andrew Ng argued that "there is no realistic way to implement a moratorium" without government intervention, which would be "anti-competitive" and "awful innovation policy." Reid Hoffman criticized the letter as "virtue signaling" that would hurt the cause by alienating the AI developer community needed to achieve safety goals.

PauseAI Movement

Aspect	Detail
Founded	May 2023 in Utrecht, Netherlands by software entrepreneur Joep Meindertsma
Goal	International moratorium on frontier AI development until safety is ensured
Structure	Network of local organizations; US chapter led by Holly Elmore, UK by Joseph Miller (Oxford PhD)
Approach	Grassroots activism, protests at AI labs (OpenAI Feb 2024, Anthropic Nov 2024), policy advocacy
Policy Asks	Global pause enforced through international treaty; democratic control over AI development
Key Actions	International protests in May 2024 timed to Seoul AI Safety Summit; protests held in San Francisco, New York, Berlin, Rome, Ottawa, London

Academic Proposals

Proposal	Scope	Mechanism
Compute Caps	Limit training compute	Hardware governance
Capability Gates	Pause at defined capability thresholds	Eval-based triggers
Conditional Pause	Pause if safety benchmarks not met	RSP-like framework
Research Moratoria	Pause specific capability research	Targeted restrictions

Implementation Challenges

Coordination Problems

Challenge	Description	Severity	Potential Solution
International Competition	US-China dynamics; neither wants to pause first	Critical	Treaty with verification
Corporate Competition	First-mover advantages; defection incentives	High	Regulatory mandate
Verification	How to confirm compliance	High	Compute monitoring
Definition	What counts as "frontier" AI	High	Clear technical thresholds

Enforcement Mechanisms

Mechanism	Feasibility	Effectiveness	Notes
Voluntary Compliance	Low	Very Low	No incentive to comply
National Regulation	Medium	Medium	Jurisdictional limits
International Treaty	Low-Medium	High if achieved	Requires major power agreement
Compute Restrictions	Medium	Medium-High	Physical infrastructure trackable
Social Pressure	Medium	Low	Insufficient against strong incentives

Unintended Consequences

Consequence	Likelihood	Severity	Mitigation
Development Displacement	High	High	International coordination
Underground Development	Medium	Very High	Compute monitoring
Safety Researcher Exodus	Medium	Medium	Continued safety funding
Competitive Disadvantage	High	Variable	Coordinated action
Delayed Benefits	High	Medium	Risk-benefit analysis

Historical Precedents

Domain	Intervention	Outcome	Lessons
Nuclear Weapons	Various moratoria and treaties	Partial success; proliferation continued	Verification essential
Human Cloning	Research moratoria	Generally effective	Narrow scope helps
Gain-of-Function	Research pause (2014-2017)	Temporary; research resumed	Pressure to resume
Recombinant DNA	Asilomar conference (1975)	Self-regulation worked initially	Community buy-in crucial
CFCs	Montreal Protocol	Highly successful	Clear harm identification

Lessons for AI Pauses

Narrow scope is more enforceable than broad moratoria
Verification mechanisms are essential for compliance
International coordination requires identifying mutual interests
Community buy-in from researchers enables voluntary compliance
Clear triggering conditions help define when restrictions apply

Scalability Assessment

Dimension	Assessment	Rationale
International Scalability	Unknown	Depends on coordination
Enforcement Scalability	Partial	Compute monitoring possible
SI Readiness	Yes (if works)	Would prevent reaching SI until prepared
Deception Robustness	N/A	External policy; doesn't address model behavior

Strategic Considerations

When Pauses Might Work

Condition	Importance	Current Status
International Agreement	Critical	Very limited
Clear Triggers	High	Undefined
Verification Methods	High	Underdeveloped
Alternative Pathway	Medium	Safety research ongoing
Industry Buy-In	Medium-High	Very low

Alternative Approaches

Alternative	Relationship to Pause	Tradeoffs
Differential Progress	Accelerate safety, not slow capabilities	Competitive with capabilities
Responsible Scaling Policies	Conditional pauses at thresholds	Voluntary; lab-controlled
Compute Governance	Indirect slowdown through resource control	More enforceable
International Coordination	Framework for coordinated pause	Slower to achieve

Quick Assessment

Dimension	Assessment	Evidence
Tractability	Low	Severe coordination and enforcement challenges; no major lab has voluntarily paused
Effectiveness	Very High (if implemented)	Would directly address timeline concerns by buying time for safety research
Neglectedness	Medium	Active advocacy (FLI, PauseAI); major gap in implementation and enforcement mechanisms
Current Maturity	Early Advocacy	FLI letter catalyzed debate but no binding commitments achieved
Time Horizon	Immediate-Long Term	Could theoretically be implemented quickly but requires international coordination
Key Proponents	FLI, PauseAI, Yoshua Bengio	Grassroots movements and prominent AI researchers
Key Opponents	Major AI Labs, Andrew Ng	Competitive dynamics and concerns about practicality

Risks Addressed

If implemented effectively, pause/moratorium would address:

Risk	Mechanism	Effectiveness
AI Development Racing Dynamics	Eliminates competitive pressure	Very High
Safety-Capability Gap	Time for safety research	Very High
Governance Lag	Time for policy development	High
Societal Preparation	Time for adaptation	High
****	Prevents deployment of unaligned systems	Very High (during pause)

Limitations

Enforcement Infeasibility: No mechanism to enforce global compliance
Competitive Dynamics: Unilateral pause disadvantages safety-conscious actors
Displacement Risk: Development may move to less cautious jurisdictions
Definition Challenges: Unclear what should be paused
Political Unreality: Insufficient political will for meaningful implementation
Temporary Nature: Pauses must eventually end; doesn't solve underlying problem

International Coordination Efforts

While a full pause has not been achieved, international efforts toward AI governance have accelerated since the 2023 open letter:

Loading diagram...

Initiative	Date	Outcome	Limitations
UK AI Safety Summit	Nov 2023	Bletchley Declaration; AI Safety Institute network launched	Non-binding; no enforcement
International AI Safety Report	2024-2025	100 AI experts contributed; comprehensive risk synthesis	Advisory only
Seoul AI Safety Summit	May 2024	16 companies signed voluntary safety commitments	No binding pause agreement
UN AI Governance	2024-2025	International Scientific Panel and Global Dialogue established	Early stage coordination

Sources & Resources

Primary Sources

Source	Type	Key Contribution
FLI Open Letter	Open Letter	Original pause proposal with 30,000+ signatories
MIT Tech Review Analysis	Journalism	Six-month retrospective on letter's impact
Bengio et al. in Science	Academic Paper	"Managing extreme AI risks amid rapid progress" (May 2024)
International AI Safety Report	Government Report	30-nation synthesis of AI safety evidence
PauseAI	Advocacy Org	Grassroots organizing and protest coordination

Key Organizations

Organization	Role	Position
Future of Life Institute	Advocacy, funding	Strong pause advocate; published open letter
PauseAI	Grassroots activism	International moratorium advocacy
GovAI	Research	Policy analysis and internationalization frameworks
Major AI Labs	Development	Opposed to pause; signed voluntary commitments only

Resource	Description
Yoshua Bengio's Blog	"Reasoning through arguments against taking AI safety seriously"
EA Forum AI Pause Debate	Community discussion of pause arguments
TIME Interview with Bengio	"We're Not Ready for AI's Risks"
Carnegie Endowment Analysis	"The AI Governance Arms Race"

References

1FLI open letterFuture of Life Institute▸

★★★☆☆

futureoflife.org

2Yoshua Bengioyoshuabengio.org▸

yoshuabengio.org

3What's changed since the "pause AI" letter six months ago?MIT Technology Review▸

★★★★☆

technologyreview.com

4Pause AI movementpauseai.info▸

pauseai.info

5Carnegie analysis warnsCarnegie Endowment▸

★★★★☆

carnegieendowment.org

6Scientific Panelpress.un.org▸

press.un.org

7Future of Life InstituteFuture of Life Institute▸

The Future of Life Institute works to guide transformative technologies like AI towards beneficial outcomes and away from large-scale risks. They engage in policy advocacy, research, education, and grantmaking to promote safe and responsible technological development.

★★★☆☆

futureoflife.org

8GovAICentre for the Governance of AI·Government▸

A research organization focused on understanding AI's societal impacts, governance challenges, and policy implications across various domains like workforce, infrastructure, and public perception.

★★★★☆

governance.ai

Pause / Moratorium