Skip to content

Safety Research & Resources

📋Page Status
Page Type:ContentStyle Guide →Standard knowledge base article
Quality:62 (Good)⚠️
Importance:72.5 (High)
Last edited:2026-01-30 (2 days ago)
Words:2.6k
Backlinks:3
Structure:
📊 13📈 1🔗 38📚 2929%Score: 14/15
LLM Summary:Comprehensive analysis of AI safety research capacity shows ~1,100 FTE researchers globally (600 technical, 500 governance) with $150-400M annual funding, representing severe under-resourcing (1:10,000 funding ratio vs capabilities). Field growing 21-24% annually but lagging capabilities growth of 30-40%, creating widening absolute gap despite tripling from ~400 FTEs in 2022.
Critical Insights (4):
  • Quant.AI safety research has only ~1,100 FTE researchers globally compared to an estimated 30,000-100,000 capabilities researchers, creating a 1:50-100 ratio that is worsening as capabilities research grows 30-40% annually versus safety's 21-25% growth.S:4.5I:5.0A:4.5
  • Quant.The spending ratio between AI capabilities and safety research is approximately 10,000:1, with capabilities investment exceeding $100 billion annually while safety research receives only $250-400M globally (0.0004% of global GDP).S:4.0I:5.0A:4.0
  • ClaimDespite rapid 25% annual growth in AI safety research, the field tripled from ~400 to ~1,100 FTEs between 2022-2025 but is still producing insufficient research pipeline with only ~200-300 new researchers entering annually through structured programs.S:3.5I:4.5A:4.5
Issues (2):
  • QualityRated 62 but structure suggests 93 (underrated by 31 points)
  • Links24 links could use <R> components
DimensionAssessmentEvidence
Total Safety Researchers≈1,100 FTEs globally (2025)AI Safety Field Growth Analysis: 600 technical + 500 governance
Annual Funding$150-400M total; $10M Coefficient Giving (2024)Coefficient Giving 2024 Report
Safety:Capabilities Ratio1:50-100 researcher ratio; 1:10,000 funding ratioStuart Russell estimates
Field Growth Rate21-24% annually (safety) vs 30-40% (capabilities)EA Forum analysis
Government Investment$160M+ combined (UK AISI: £240M, US AISI: $10M)UK AISI grants, NIST budget
Training Pipeline≈300 new researchers/year via structured programsMATS (98 scholars), SPAR (50+), ERA (30+)
Industry Safety IndexD average for existential safety across all labsFLI AI Safety Index 2025

This page tracks the size, growth, and resource allocation of the AI safety research field. Understanding these metrics helps assess whether safety research is keeping pace with capabilities development and identify critical capacity gaps. The analysis encompasses researcher headcount, funding flows, publication trends, and educational programs.

Key finding: Despite rapid growth, AI safety research remains severely under-resourced relative to capabilities development, with spending ratios estimated at 1:10,000 or worse. The field has tripled from ~400 to ~1,100 FTEs (2022-2025) but capabilities research is growing faster, creating a widening absolute gap. Current safety funding represents just 0.0004% of global GDP, while AI capabilities investment exceeds $100 billion annually. This creates significant questions about whether AI safety research can develop adequate solutions before transformative AI capabilities emerge.

DimensionAssessmentEvidenceTrend
Researcher ShortageCritical1:50-100 safety:capabilities ratioWorsening
Funding GapSevere1:10,000 spending ratioStable disparity
Experience GapHighMedian 2-5 years experienceSlowly improving
Growth Rate MismatchConcerning21% vs 30-40% annual growthGap widening

Loading diagram...
CategoryCountOrganizationsGrowth Rate
Technical AI Safety≈600 FTEs68 active orgs21% annually
AI Governance/Policy≈500 FTEsVarious30% annually
Total Safety Research≈1,100 FTEs70+ orgs25% annually

Data source: AI Safety Field Growth Analysis 2025 tracking organizations explicitly branded as “AI safety.”

Key limitations: 80,000 Hours estimates “several thousand people” work on major AI risks when including researchers at major labs and academia, suggesting significant undercounting of part-time and embedded safety researchers.

Top technical research areas by organization count:

  1. Miscellaneous technical AI safety research
  2. LLM safety
  3. Interpretability
  4. Alignment research

Historical growth trajectory:

  • 2022: ~400 FTE researchers total
  • 2023: ~650 FTE researchers
  • 2024: ~900 FTE researchers
  • 2025: ~1,100 FTE researchers

This represents consistent 25%+ annual growth, but still lags behind estimated capabilities research expansion of 30-40% annually.


Annual Safety Research Funding (2024-2025)

Section titled “Annual Safety Research Funding (2024-2025)”
Funding SourceAmountFocus AreaReliability
Coefficient Giving≈$10M (2024); $10M RFP (2025)Technical safety (21 research areas), governanceHigh
Long-Term Future Fund≈$1-10M annuallyIndividual grants, upskillingMedium
Government Programs≈$160M+UK AISI (£240M), US AISI ($10M), Canada ($10M)Growing
Corporate LabsUndisclosedInternal safety teamsUnknown
Total Estimated$150-400MGlobal safety researchMedium confidence

Coefficient Giving context: Since 2017, Coefficient Giving (then Open Philanthropy) has donated ≈$136 million to AI safety (~12% of their $1.8B total grants). They acknowledged their 2024 spending rate was “too slow” and are “more aggressively expanding support for technical AI safety work.” Their 2025 RFP covers 21 research directions including adversarial testing, model transparency, and theoretical alignment.

Country/RegionProgramFundingKey InitiativesTimeline
United KingdomUK AI Security Institute£240M total£15M Alignment Project, £8.5M Systemic Safety Grants, £200K Challenge Fund2023+
United StatesUS AISI (renamed CAISI 2025)$10M (chronically underfunded)Model evaluation partnerships with Anthropic/OpenAI2024+
CanadaCanada AISI$10MResearch coordination2024+
European UnionAI Act implementation€100M+Regulatory infrastructure2024+

Note: The UK-US AI Safety Institutes signed a landmark agreement in 2024 to jointly test advanced AI models, share research insights, and enable expert talent transfers. However, US funding remains substantially lower than UK investment—the NIST budget that hosts AISI has faced congressional budget cuts rather than the expansion requested by the Biden administration.

Critical disparity metrics:

  • 10,000:1 ratio of capabilities to safety investment (Stuart Russell, UC Berkeley)
  • Companies spend more than $100 billion building AGI vs ≈$10 million philanthropic safety research annually
  • AI safety funding: 0.0004% of global GDP vs $131.5B in AI startup VC funding (2024)
  • Only 2% of AI publications concern safety issues despite 312% growth in safety research (2018-2023)
  • External safety organizations operate on budgets smaller than a frontier lab’s daily burn

Capability researcher growth comparison (AI Safety Field Growth Analysis 2025):

MetricSafety FieldCapabilities FieldGap
Annual growth rate21-24%30-40%Widening
OpenAI headcountN/A300 → 3,000 (2021-2025)10x growth
Anthropic, DeepMindN/AEach grown more than 3xRapid expansion
ML papers per year≈45,000 safety-related (2018-2023)Doubles every 2 yearsExponential

For context: Global philanthropic climate funding reaches $1-15 billion annually, making climate funding 20-40x larger than AI safety funding. Prominent AI safety advocates recommend increasing safety investment to at least 30% of compute resources, a level far above current allocations.


Major alignment research developments:

Research AreaNotable 2024-2025 PapersImpact
Alignment Foundations”AI Alignment: A Comprehensive Survey” (RICE framework)Comprehensive taxonomy
Mechanistic Interpretability”Mechanistic Interpretability Benchmark (MIB)“Standardized evaluation
Safety BenchmarksWMDP Benchmark (ICML 2024)Dangerous capability assessment
Training Methods”Is DPO Superior to PPO for LLM Alignment?”Training optimization

Industry research contributions:

  • Anthropic: Circuit tracing research revealing Claude’s “shared conceptual space” (March 2025)
  • Google DeepMind: Announced deprioritizing sparse autoencoders (March 2025)
  • CAIS: Supported 77 safety papers through compute cluster (2024)

Field debates: Intensified discussion about mechanistic interpretability value, with Dario Amodei advocating focus while other labs shift priorities.

Positive signals:

  • Research “moving beyond raw performance to explainability, alignment, legal and ethical robustness”
  • Standardized benchmarks emerging (MIB, WMDP)
  • Industry-academic collaboration increasing

Concerning signals:

FLI AI Safety Index: Company Comparison (Winter 2025)

Section titled “FLI AI Safety Index: Company Comparison (Winter 2025)”

The Future of Life Institute AI Safety Index evaluates leading AI companies across six domains using 33 indicators. Scores use US GPA system (A+ to F).

CompanyOverallRisk AssessmentCurrent HarmsSafety FrameworkExistential SafetyGovernanceInformation Sharing
AnthropicB-BB+BDBB-
OpenAIC+B-BB-DC+C
Google DeepMindC+B-B-B-DCC
xAID+DCDD-DD
MetaDD+C+D-D-D-D
Zhipu AID-DCD-FD-D-
DeepSeekD-D-C-D-FFD-

Key finding: No company achieved above a D in Existential Safety, indicating industry-wide structural failure to prevent catastrophic misuse or loss of control. The top three performers (Anthropic, OpenAI, DeepMind) show substantially stronger practices than others, particularly in risk assessment and safety frameworks.


ProgramFundingDurationFocus
Vitalik Buterin PhD Fellowship$10K/year + tuition5 yearsAI safety PhD research
Google PhD Fellowship$85K/yearVariableAI research including safety
Global AI Safety FellowshipUp to $30K6 monthsCareer transitions
Anthropic Fellows Program$2,100/weekFlexibleMid-career transitions
ProgramAnnual CapacityTarget AudienceOutcomesSupport
MATS98 scholars (Summer 2025)Aspiring safety researchers80% now work in AI safety; 10% co-founded startups$15K stipend, $12K compute, housing
SPAR50+ participantsUndergraduate to professionalResearch publicationsMentorship, resources
ERA Fellowship30+ fellowsEarly-career researchersCareer transitionsFunding, network
LASR LabsVariableResearch transitionsLab placementsProject-based

MATS program details: The MATS Summer 2025 cohort supported 98 scholars with 57 mentors across interpretability, governance, and security research tracks. Alumni outcomes show ~80% continue in AI safety/security roles, with ~75% continuing in fully-funded 6-12 month extensions. Notable alumni have published award-winning papers (ACL 2024 Outstanding Paper) and joined frontier labs like Anthropic. Program satisfaction averages 9.4/10.

Estimated field pipeline: ~300 new safety researchers entering annually through structured programs, plus unknown numbers through academic and industry pathways.


Major AI Conference Attendance (2024-2025)

Section titled “Major AI Conference Attendance (2024-2025)”
ConferenceTotal SubmissionsAttendanceAI Safety ContentGrowth
NeurIPS 202521,575 valid submissions (5,290 accepted, 24.5%)16,000+8 safety-focused social sessions61% submission increase
NeurIPS 202419,756 participants≈16,000Safety workshops, CAIS papers27% increase
ICML 20249,095 participants9,095”Next Generation of AI Safety” workshop15% increase
ICLR 2024≈8,000 participants≈8,000Alignment research track12% increase

NeurIPS 2025 context: The conference saw a 61% increase in submissions over 2024, supported by 20,518 reviewers, 1,663 area chairs, and 199 senior area chairs. This massive growth reflects the global surge in AI research productivity, though safety-specific research remains a small fraction of total submissions.

Safety-specific events:

  • CAIS online course: 240 participants (2024)
  • AI safety conference workshops and socials organized by multiple organizations
  • NeurIPS 2025 split between Mexico City and Copenhagen due to capacity constraints

Positive trends:

  • Safety workshops becoming standard at major AI conferences
  • Industry participation in safety research increasing
  • Graduate programs adding AI safety coursework

Infrastructure constraints:

  • Major conferences approaching venue capacity limits
  • Competition for safety researcher talent intensifying
  • Funding concentration creating bottlenecks

MetricCurrent State (2025)Current GrowthRequired GrowthGap Assessment
Safety researchers≈1,100 FTEs21-24% annually50%+ (to catch up)Critical: widening
Safety funding$150-400M≈25% annually100%+ (recommended 30% of compute)Severe
Safety publications≈2% of AI papers≈20% annually (312% growth 2018-2023)UnknownModerate
Training pipeline≈300/yearGrowing≈1,000/year neededSignificant

Based on current exponential growth models from the AI Safety Field Growth Analysis 2025:

Optimistic scenario (current 21-24% growth continues):

  • ~2,500-3,000 FTE safety researchers by 2030 (extrapolating from current 1,100)
  • ≈$100M-1B annual safety funding by 2028
  • Mature graduate programs producing 500+ researchers annually
  • UK AISI Alignment Project produces breakthrough research

Concerning scenario (capabilities growth accelerates to 50%+):

  • Safety research remains under 5% of total AI research
  • Racing dynamics intensify as AGI timelines compress
  • 30-40% capabilities growth vs 21-24% safety growth creates widening absolute gap
  • External safety organizations continue operating on budgets smaller than frontier lab daily burn

Capability researcher count: No comprehensive database exists for AI capabilities researchers. Estimates suggest 30,000-100,000 globally based on:

  • OpenAI growth: 300→3,000 employees (2021-2025)
  • Similar expansion at Anthropic, DeepMind
  • ML conference attendance doubling every 2-3 years

Industry safety spending: Most AI labs don’t disclose safety vs capabilities budget breakdowns. Known examples:

  • IBM: 2.9%→4.6% of AI budgets (2022-2024)
  • OpenAI: Super-alignment team disbanded (May 2024)
  • Anthropic: Constitutional AI research ongoing but budget undisclosed

Field size adequacy:

  • Optimists: Current growth sufficient if focused on highest-impact research
  • Pessimists: Need 10x more researchers given AI risk timeline

Research prioritization:

  • Technical focus: Emphasize interpretability, alignment
  • Governance focus: Prioritize policy interventions, coordination

Funding allocation:

  • Large grants to established organizations vs distributed funding for diverse approaches
  • Academic vs industry vs independent researcher support ratios

MetricData QualityPrimary LimitationsImprovement Needs
FTE researchersMediumUndercounts independents, part-time contributorsComprehensive workforce survey
Total fundingMediumMany corporate/government grants undisclosedDisclosure requirements
Spending ratiosLowLabs don’t publish safety budget breakdownsIndustry transparency standards
Publication trendsMediumNo centralized safety research databaseStandardized taxonomy and tracking
Experience levelsVery LowNo systematic demographic data collectionRegular field census
Researcher ratiosLowNo capability researcher baseline countComprehensive AI workforce analysis

Most critical data gaps:

  1. Industry safety spending: Mandatory disclosure of safety vs capabilities R&D budgets
  2. Researcher demographics: Experience, background, career transition patterns
  3. Research impact assessment: Citation analysis and influence tracking for safety work
  4. International coordination: Non-English language safety research and global South participation


Last updated: January 30, 2026

Note: This analysis synthesizes data from multiple sources with varying quality and coverage. Quantitative estimates should be interpreted as order-of-magnitude indicators rather than precise counts. The field would benefit significantly from standardized data collection and reporting practices.