Safety Spending at Scale
Safety Spending at Scale
Models what AI safety spending could accomplish at different budget levels from \$1B to \$50B+/year. Current global safety spending (~\$500M-1B/year) is 100-600x below capabilities investment. At \$5B/year, could fund 5,000+ dedicated safety researchers, comprehensive interpretability programs, and independent evaluation infrastructure. Key finding: absorptive capacity is the binding constraint below \$10B/year—the field cannot productively absorb unlimited funding without growing the researcher pipeline (current ~1,000 qualified safety researchers globally). Above \$10B/year, institutional capacity and research direction clarity become primary constraints. Provides concrete portfolio recommendations at each funding level.
Overview
Current global AI safety spending is approximately $700M-1.25B per year, while AI capabilities investment exceeds $100B annually—a ratio of roughly 100:1 to 150:1, though these figures depend heavily on how "safety spending" is defined.1 The AI Safety Research Value Model suggests scaling safety funding 3-10x to $2-5B/year. The Pre-TAI Capital Deployment analysis shows that frontier labs alone may spend $100-300B+ over the coming years. This page examines what different levels of safety spending—$5B, $10B, or $50B annually—could accomplish given current field constraints.
This analysis models the absorptive capacity problem: the gap between theoretical safety spending and what the field can productively deploy. It provides portfolio recommendations at different funding levels and identifies bottlenecks that constrain effective scaling.
Central finding: Under these assumptions, below approximately $10B/year, the primary constraint is the researcher talent pipeline. Above that level, institutional capacity, research direction clarity, and the fundamental difficulty of the alignment problem become dominant constraints. This suggests that expanding safety spending requires parallel investment in talent development rather than funding alone.
Current Baseline
Global AI Safety Spending (2025 Estimates)
| Source | Annual Spending | Growth Rate | Primary Focus |
|---|---|---|---|
| Frontier AI labs (internal) | $400-700M | +40-50%/year | Interpretability, RLHF, evaluations, red-teaming |
| Philanthropic funders | $150-250M | +20-30%/year | Academic research, field building, policy |
| Government | $100-200M | +50-100%/year | Standards (NIST), evaluation (UK AISI), military |
| Academia (dedicated safety) | $50-100M | +15-20%/year | Theoretical alignment, robustness, fairness |
| Total | $700M-1.25B | +35-50%/year |
Current Safety Research Workforce
| Level | Estimated Count | Average Compensation | Total Cost | Notes |
|---|---|---|---|---|
| Senior safety researchers | 150-300 | $600K-1.5M | $200-400M | At frontier labs + top academic positions |
| Mid-level safety researchers | 500-1,000 | $300K-600K | $200-500M | Labs, research orgs, academia |
| Junior/entry-level | 1,000-2,000 | $100K-300K | $150-400M | PhD students, postdocs, early career |
| Adjacent researchers (ML safety-relevant) | 2,000-5,000 | $200K-500K | Not counted | Working on safety-adjacent problems |
| Total dedicated safety workforce | ≈2,000-3,500 | $550M-1.3B |
Approximately 2,000-3,500 people globally work full-time on AI safety research as of 2025.2
Absorptive Capacity Analysis
What Limits Safety Spending?
The concept of "absorptive capacity" models how much funding can be productively deployed at different scales. The binding constraint varies by funding level:
Binding Constraints by Funding Level
| Funding Level | Binding Constraint | Estimated Absorptive Rate | Key Action Required |
|---|---|---|---|
| $1-2B/yr | Talent (not enough researchers) | 70-85% | Build pipeline; expand training programs |
| $2-5B/yr | Talent + infrastructure | 50-70% | Create new labs; fund compute access |
| $5-10B/yr | Institutional capacity | 40-60% | Build organizations; develop research agendas |
| $10-30B/yr | Research direction clarity | 30-50% | Need fundamental breakthroughs to guide resource allocation |
| $30-50B+/yr | Problem difficulty | 20-40% | Alignment difficulty may limit returns to additional resources |
Note: "Absorptive rate" estimates represent the approximate fraction of funding that could be deployed on high-value safety research under current field constraints, based on researcher availability, institutional capacity, and research direction clarity. These are model estimates rather than empirical measurements.
Portfolio Recommendations by Funding Level
Level 1: $1-2B/year (2-4x Current)
Primary objective: Expand the researcher pipeline while maintaining research quality.
This funding level could be absorbed with moderate efficiency given current field constraints.
| Category | Allocation | Annual Spend | What It Funds |
|---|---|---|---|
| Expand existing orgs | 30% | $300-600M | Double safety teams at Anthropic, DeepMind, OpenAI; expand ARC Evaluations, MIRI, Redwood Research |
| Talent pipeline | 25% | $250-500M | PhD fellowships (500+), postdoc programs, bootcamps, career transitions |
| Academic programs | 20% | $200-400M | 20-30 university safety research centers; endowed chairs |
| Evaluations & red-teaming | 15% | $150-300M | Independent eval orgs; bug bounties; adversarial testing infrastructure |
| Governance research | 10% | $100-200M | Policy research; regulatory frameworks; international coordination |
Projected outcomes (3-5 year horizon):
- Safety researcher workforce: 2,500-3,500 → 5,000-7,000
- Independent evaluation capacity: 5-10x current levels
- Academic safety programs: 10-20 → 50-80 universities
- Capabilities-to-safety spending ratio: 100:1 → 50:1
Level 2: $5-10B/year (5-10x Current)
Primary objective: Build institutional safety research capacity independent of capability-focused labs.
At this level, safety research could develop organizational infrastructure comparable to other mature research disciplines.
| Category | Allocation | Annual Spend | What It Funds |
|---|---|---|---|
| Dedicated safety labs | 25% | $1.25-2.5B | 3-5 new independent safety research institutes at scale |
| Safety compute | 20% | $1-2B | Dedicated compute clusters for safety research; model access |
| Expanded talent pipeline | 15% | $750M-1.5B | 1,000+ new PhD positions; global training programs |
| Interpretability at scale | 15% | $750M-1.5B | Large-scale mechanistic interpretability; automated tools |
| Evaluation infrastructure | 10% | $500M-1B | National-scale evaluation facilities; continuous monitoring |
| Governance & institutions | 10% | $500M-1B | International coordination mechanisms; regulatory capacity building |
| Exploratory research | 5% | $250-500M | High-variance fundamental research; novel approaches |
Projected outcomes (3-5 year horizon):
- Safety researcher workforce: 10,000-15,000
- Multiple independent labs with frontier model access
- Continuous safety monitoring systems for deployed models
- Established academic discipline with standardized curricula
- Capabilities-to-safety spending ratio: 20:1 → 10:1
Level 3: $10-30B/year (Capabilities-Scale Funding)
Primary objective: Achieve resource parity with capabilities research in key safety domains.
At this scale, efficiency concerns increase substantially. The constraint shifts from researcher availability to clarity about productive research directions.
| Category | Allocation | Annual Spend | What It Funds |
|---|---|---|---|
| Alignment R&D | 25% | $2.5-7.5B | Large-scale alignment experiments; parallel research programs |
| Safety infrastructure | 20% | $2-6B | Dedicated safety compute; monitoring systems; secure facilities |
| Human capital | 15% | $1.5-4.5B | Global researcher pipeline; competitive compensation |
| Interpretability | 15% | $1.5-4.5B | Automated interpretability tools; comprehensive model understanding |
| Governance & policy | 10% | $1-3B | International institutions; regulatory implementation; standards |
| Applied safety | 10% | $1-3B | Deployment safety; incident response; societal resilience |
| Exploratory approaches | 5% | $500M-1.5B | High-risk/high-reward approaches; formal verification at scale |
Level 4: $30-50B+/year (Maximum Plausible Scale)
At this level, safety investment approaches capabilities investment levels. The primary uncertainty shifts to whether additional resources yield proportional safety improvements.
Key question: If alignment is fundamentally difficult—requiring conceptual breakthroughs rather than engineering effort—then $50B may not provide substantially more value than $10B. Additional funding is most valuable if:
- Safety research benefits from parallelizable investigations across multiple approaches
- Compute-intensive experiments are resource-constrained rather than idea-constrained
- Applied safety engineering (monitoring, evaluation, deployment) can scale with model deployment
Highest-value applications at this scale:
- Comprehensive safety infrastructure (monitoring, evaluation, incident response) scaling with deployment
- Large-scale interpretability programs comparable to genome sequencing projects
- Redundant, independent safety research programs to increase probability of breakthroughs
- International safety institutions with enforcement capacity
The Talent Pipeline Problem
The current pipeline produces approximately 200-500 new safety researchers per year (see AI Talent Market Dynamics for detailed pipeline analysis). To productively absorb $5-10B/year would require 10,000-15,000 researchers—implying a 3-5x increase in the pipeline sustained over several years.
Pipeline Expansion Strategies
| Strategy | Annual Cost | Timeline | Estimated Scale Impact | Quality Considerations |
|---|---|---|---|---|
| PhD fellowships ($100K/year each) | $500M for 1,000 positions | 4-6 years | High | Low if selective |
| Postdoc bridge programs | $200M/year | 1-3 years | Medium | Low |
| Industry → safety career transitions | $100M/year | 1-2 years | Medium | Medium |
| University center creation | $50M each, 20+ centers | 3-5 years | High | Low-Medium |
| Bootcamps/intensive programs | $10M/year for 500 seats | 3-6 months | Low-Medium | Medium-High |
| International programs (non-US/UK) | $200M/year | 2-4 years | High | Medium |
| Competitive compensation matching | $300M+/year premium | Immediate | Medium | Low |
The Compensation Gap
| Role | Industry (Lab) | Safety Research Org | Academic | Gap |
|---|---|---|---|---|
| Senior Researcher | $800K-2M | $250K-600K | $120K-250K | 2-8x |
| Mid-level | $400K-800K | $150K-300K | $80K-150K | 2-5x |
| Junior | $200K-400K | $80K-150K | $40K-80K | 2-5x |
Reducing this compensation gap by 50% for safety researchers would cost $300-500M/year across the field, and could increase talent flow into safety roles.
Historical Analogies for Scaling Safety Research
Nuclear Safety (1950s-1980s)
Nuclear weapons and power development provides the most direct historical parallel for scaling safety research in response to catastrophic risk.
| Period | Annual Safety Spend (2024 $) | Researchers | Key Development | Ratio to Capabilities Spend |
|---|---|---|---|---|
| 1945-1955 | $50-200M | Hundreds | Basic radiation protection standards | ≈1:100 |
| 1955-1970 | $500M-2B | Thousands | Reactor safety frameworks; test ban treaty | ≈1:20 |
| 1970-1985 | $2-5B | ≈10,000 | NRC establishment; comprehensive regulatory standards | ≈1:10 |
| Post-TMI (1979+) | $5-10B | 15,000+ | Containment standards; probabilistic risk assessment | ≈1:5 |
Key patterns:
- Safety investment scaled substantially after the Three Mile Island incident (1979)
- The safety research apparatus required 20-30 years to develop institutional maturity
- Safety spending reached 10-20% of capabilities investment but never exceeded it
- Institutional capacity (NRC, IAEA) development was as important as research funding
Pharmaceutical Safety
| Metric | Pharma Safety | AI Safety (Current) | AI Safety (Modeled) |
|---|---|---|---|
| Safety as % of R&D | 15-25% | 1-3% | 5-10% (proposed) |
| Regulatory bodies | FDA, EMA, many national agencies | NIST AISI, UK AISI (nascent) | To be determined |
| Independent testing | Required pre-deployment | Voluntary, inconsistent | To be determined |
| Workforce | ≈50,000 in safety/regulatory | ≈3,000 | 10,000-30,000 (projected) |
| Time to maturity | ≈50 years (1938-1990s) | ≈5 years so far | Unknown |
Categories of Safety Work
Different types of safety work address different categories of risk. Understanding this distribution is important for evaluating spending effectiveness:
Technical Alignment Research
Focus: Ensuring AI systems reliably pursue intended objectives
Examples: Scalable oversight, interpretability, value learning
Primary risk addressed: Deceptive alignment, scheming, goal misgeneralization
Estimated current spending: $300-500M/year
Deployment Safety
Focus: Preventing harm from deployed systems
Examples: Content moderation, robustness testing, red-teaming
Primary risk addressed: Misuse, accidents, unintended consequences
Estimated current spending: $200-400M/year
Evaluation and Monitoring
Focus: Detecting dangerous capabilities and behaviors
Examples: ARC Evaluations, METR, Responsible Scaling Policies
Primary risk addressed: Surprise capability gains, deployment of unsafe systems
Estimated current spending: $100-200M/year
Governance and Policy
Focus: Institutional mechanisms for managing AI risk
Examples: International coordination, regulatory frameworks, standards development
Primary risk addressed: Race dynamics, inadequate oversight, coordination failures
Estimated current spending: $100-200M/year
Public Communication and Trust
Focus: Building public understanding and institutional legitimacy
Examples: Safety announcements, transparency reports, public engagement
Primary risk addressed: Loss of public trust, regulatory overreach, insufficient public input
Estimated current spending: $50-150M/year
Some work announced as "safety" primarily serves branding or regulatory compliance rather than reducing technical or catastrophic risk. Indicators that spending may not reduce existential risk include:
- Safety announcements timed with product launches rather than research milestones
- Focus on content moderation or PR-relevant metrics rather than catastrophic risk
- Lack of independent oversight or publication of negative results
- Evaluation programs that test user-facing features rather than dangerous capabilities
The OpenAI Foundation page documents related concerns in the philanthropic context. Whether public communication and trust-building constitute legitimate safety investments depends on whether institutional legitimacy is necessary for effective safety regulation—a question with reasonable disagreement.
Spending Efficiency Concerns
At larger scales, several failure modes become more likely:
| Failure Mode | Risk Level | Example | Mitigation |
|---|---|---|---|
| Duplicated effort | High at >$5B | Multiple teams solving identical problems | Coordination bodies; shared infrastructure |
| Hiring for headcount targets | Very High at >$2B | Lowering standards to fill positions | Maintain hiring standards; accept slower growth |
| Infrastructure without direction | High at >$10B | Compute purchases without clear research questions | Fund researchers first, compute second |
| Institutional overhead | Medium at all levels | Administrative costs consuming 30-40% of budget | Lean organizations; direct grants |
| Misdirected research | High at all levels | Funding approaches unlikely to address core risks | Diverse portfolio; regular evaluation |
Counterarguments and Limitations
Cases Where More Spending Might Not Help
- Fundamental difficulty: If alignment requires conceptual breakthroughs rather than engineering effort, scaling spending may not proportionally reduce risk
- Information hazards: Publishing certain safety research could accelerate capabilities or provide blueprints for misuse
- Researcher quality dilution: Rapid expansion may lower average researcher quality below productivity thresholds
- Opportunity costs: Safety spending competes with other existential risk reduction (biosecurity, nuclear security)
Measurement Challenges
Evaluating safety research effectiveness faces fundamental difficulties:
- Long feedback loops (decades until deployment outcomes are known)
- Counterfactual dependence (success means preventing events that don't occur)
- Dual-use research (safety work that also advances capabilities)
- Publication bias (positive results published more than negative)
Geographic Distribution
Current safety research is concentrated in the US (≈50%), UK (≈20%), and Western Europe (≈15%), with limited presence in Asia, Latin America, and Africa. This concentration creates:
- Single-points-of-failure if one jurisdiction restricts safety research
- Limited cultural and technical diversity in safety approaches
- Coordination challenges for international AI governance
- Workforce constraints as non-US researchers face visa limitations
Timeline Sensitivity
The analysis above assumes 5-15 year timelines to transformative AI. If timelines are shorter (≈3 years), the emphasis shifts toward:
- Deployment safety and monitoring over long-term alignment research
- Existing researcher productivity over pipeline building
- Government coordination over academic institution building
If timelines are longer (≈20+ years), emphasis shifts toward:
- Fundamental theoretical research
- Building sustainable academic institutions
- International norm development
Recommendations
For Frontier AI Labs
- Publish safety spending — Verifiable figures in annual reports would enable external assessment
- Fund independent safety research — External researchers provide complementary perspectives
- Provide model access to safety researchers — Safety research requires access to frontier systems
- Consider minimum safety allocation commitments — Some labs (e.g., Anthropic) allocate 5-8% of budgets to safety; whether this should be an industry norm remains debated
For Philanthropic Funders
- Prioritize pipeline over projects at current funding levels—the bottleneck is researchers, not ideas
- Fund at competitive salaries to reduce talent flow from safety to capabilities roles
- Support institutional development — Safety research requires organizations, not just individual grants
- Invest in research direction clarity through field surveys, theory development, and agenda-setting
For Governments
- Fund public safety evaluation infrastructure — Independent testing capacity comparable to FDA drug approval
- Invest in academic safety programs through research grants, center funding, and fellowship programs
- Consider safety spending disclosure requirements — Transparency as precondition for informed regulation
- Support international coordination mechanisms to enable consistent standards
Key Uncertainties
| Uncertainty | Impact on Analysis | Resolution Timeline |
|---|---|---|
| Is alignment fundamentally difficult? | If yes, diminishing returns above $5-10B; if no, resources can solve it | Unknown |
| Will labs voluntarily increase safety spend? | Determines need for external mandates | 1-3 years |
| Can the talent pipeline scale 5-10x? | Determines achievable scale of safety research | 3-5 years |
| Will safety insights transfer across architectures? | Affects long-term value of current investment | Ongoing |
| How much safety compute is needed? | Determines whether infrastructure or talent is primary bottleneck | 2-3 years |
Sources
Footnotes
-
Estimates based on published safety team sizes from <EntityLink id="E22" name="anthropic">Anthropic</EntityLink>, <En... — Estimates based on published safety team sizes from <EntityLink id="E22" name="anthropic">Anthropic</EntityLink>, <EntityLink id="E218" name="openai">OpenAI</EntityLink>, and <EntityLink id="E98" name="deepmind">Google DeepMind</EntityLink> public reporting, and philanthropic grant databases including <EntityLink id="E552" name="open-philanthropy">Open Philanthropy</EntityLink> grants database (2024). ↩
-
Citation rc-138d (data unavailable — rebuild with wiki-server access) ↩