AI Safety Organizations (Overview)

Overview

The AI safety organizational landscape spans dedicated alignment research labs, policy think tanks, advocacy groups, and field-building institutions. These organizations aim to reduce catastrophic and existential risks from advanced AI systems through technical research, governance advocacy, talent development, and public engagement.

Funding is heavily concentrated through a small number of major funders, most prominently Coefficient GivingOrganizationCoefficient GivingCoefficient Giving (formerly Open Philanthropy) has directed \$4B+ in grants since 2014, including \$336M to AI safety (~60% of external funding). The organization spent ~\$50M on AI safety in 2024...Quality: 55/100, which has provided grants to the majority of organizations listed on this page. This concentration produces a relatively coordinated funding environment, with most grantees sharing compatible research agendas and norms, while also reducing diversification of funding sources across the field.

Note: The AI safety organizational landscape evolves rapidly. Headcount, budget, and focus area descriptions reflect available information as of mid-2025 and may not capture recent changes. Check individual entity pages for the most current details.

Alignment Research Labs

Dedicated organizations conducting technical AI safety research:

• ARC (Alignment Research Center)OrganizationAlignment Research CenterComprehensive reference page on ARC (Alignment Research Center), covering its evolution from a dual theory/evals organization to ARC Theory (3 permanent researchers) plus the METR spin-out (Decembe...Quality: 57/100: Founded by Paul ChristianoPersonPaul ChristianoComprehensive biography of Paul Christiano documenting his technical contributions (IDA, debate, scalable oversight), risk assessment (~10-20% P(doom), AGI 2030s-2040s), and evolution from higher o...Quality: 39/100; focuses on alignment evaluation and theoretical alignment research
• METROrganizationMETRMETR conducts pre-deployment dangerous capability evaluations for frontier AI labs (OpenAI, Anthropic, Google DeepMind), testing autonomous replication, cybersecurity, CBRN, and manipulation capabi...Quality: 66/100: Evaluates dangerous capabilities in frontier AI models; spun out of ARC
• Apollo ResearchOrganizationApollo ResearchApollo Research demonstrated in December 2024 that all six tested frontier models (including o1, Claude 3.5 Sonnet, Gemini 1.5 Pro) engage in scheming behaviors, with o1 maintaining deception in ov...Quality: 58/100: Focuses on detecting and understanding deceptive AI behavior, including schemingRiskSchemingScheming—strategic AI deception during training—has transitioned from theoretical concern to observed behavior across all major frontier models (o1: 37% alignment faking, Claude: 14% harmful compli...Quality: 74/100 evaluations
• Redwood ResearchOrganizationRedwood ResearchA nonprofit AI safety and security research organization founded in 2021, known for pioneering AI Control research, developing causal scrubbing interpretability methods, and conducting landmark ali...Quality: 78/100: Alignment research lab working on interpretabilitySafety AgendaInterpretabilityMechanistic interpretability has extracted 34M+ interpretable features from Claude 3 Sonnet with 90% automated labeling accuracy and demonstrated 75-85% success in causal validation, though less th...Quality: 66/100, adversarial trainingApproachAdversarial TrainingAdversarial training, universally adopted at frontier labs with \$10-150M/year investment, improves robustness to known attacks but creates an arms race dynamic and provides no protection against m...Quality: 58/100, and AI controlSafety AgendaAI ControlAI Control is a defensive safety approach that maintains control over potentially misaligned AI through monitoring, containment, and redundancy, offering 40-60% catastrophic risk reduction if align...Quality: 75/100
• ConjectureOrganizationConjectureConjecture is a 30-40 person London-based AI safety org founded 2022, pursuing Cognitive Emulation (CoEm) - building interpretable AI from ground-up rather than aligning LLMs - with \$30M+ Series A...Quality: 37/100: Alignment research and product company based in London
• FAR AIOrganizationFAR AIFAR AI is an AI safety research nonprofit founded in July 2022 by Adam Gleave (CEO) and Karl Berzins (Co-founder & President). Based in Berkeley, California, the organization conducts technical res...Quality: 76/100: Researches robustness, adversarial attacks, and alignment failures in AI systems
• Palisade ResearchOrganizationPalisade ResearchPalisade Research is a 2023-founded nonprofit conducting empirical research on AI shutdown resistance and autonomous hacking capabilities, with notable findings that some frontier models resist shu...Quality: 65/100: Focuses on practical AI safety evaluation and red-teaming
• Seldon LabOrganizationSeldon LabSeldon Lab is a San Francisco-based AI safety accelerator founded in early 2025 that combines research publication with startup investment, claiming early success with portfolio companies raising \...Quality: 45/100: Works on alignment approaches and safety evaluations
• GoodfireOrganizationGoodfireGoodfire is a well-funded AI interpretability startup valued at \$1.25B (Feb 2026) developing mechanistic interpretability tools like Ember API to make neural networks more transparent and steerabl...Quality: 68/100: Interpretability-focused startup building tools for understanding neural networks
• MIRI (Machine Intelligence Research Institute)OrganizationMachine Intelligence Research InstituteComprehensive organizational history documenting MIRI's trajectory from pioneering AI safety research (2000-2020) to policy advocacy after acknowledging research failure, with detailed financial da...Quality: 50/100: Pioneer in AI alignmentApproachAI AlignmentComprehensive review of AI alignment approaches finding current methods (RLHF, Constitutional AI) show 75%+ effectiveness on measurable safety metrics for existing systems but face critical scalabi...Quality: 91/100 theory; founded 2000

Policy and Governance Organizations

Think tanks and research centers focused on AI governance and policy:

• GovAIOrganizationGovAIGovAI is an AI policy research organization with ~15-20 staff, funded primarily by Coefficient Giving (\$1.8M+ in 2023-2024), that has trained 100+ governance researchers through fellowships and cu...Quality: 43/100: Research center focused on AI governance based at Oxford
• CSET (Center for Security and Emerging Technology)OrganizationCSET (Center for Security and Emerging Technology)CSET is a \$100M+ Georgetown center with 50+ staff conducting data-driven AI policy research, particularly on U.S.-China competition and export controls. The center conducts hundreds of annual gove...Quality: 43/100: Georgetown think tank producing policy-relevant research on AI and emerging technologies
• CSER (Centre for the Study of Existential Risk)OrganizationCSER (Centre for the Study of Existential Risk)CSER is a Cambridge-based existential risk research centre founded in 2012, now funded at ~\$1M+ annually from FLI and other sources, producing 24+ publications in 2022 across AI safety, biosecurit...Quality: 58/100: Cambridge-based research center studying existential risks including from AI
• Secure AI ProjectOrganizationSecure AI ProjectPolicy advocacy organization founded ~2022-2023 by Nick Beckstead focusing on legislative requirements for AI safety protocols, whistleblower protections, and risk mitigation incentives. Rated high...Quality: 47/100: Advocacy organization focused on AI safety policy
• ControlAIOrganizationControlAIControlAI is a UK-based advocacy organization that has achieved notable policy engagement success (briefing 150+ lawmakers, securing support from 100+ UK parliamentarians) while promoting direct in...Quality: 63/100: Advocacy organization pushing for stronger AI regulation and safety standards
• Pause AIOrganizationPause AIPause AI is a grassroots advocacy movement founded May 2023 calling for international pause on frontier AI development until safety proven, growing to multi-continental network but achieving zero d...Quality: 59/100: Grassroots advocacy movement calling for a pause on frontier AI development
• Frontier Model ForumOrganizationFrontier Model ForumThe Frontier Model Forum represents the AI industry's primary self-governance initiative for frontier AI safety, establishing frameworks and funding research, but faces fundamental criticisms about...Quality: 58/100: Industry-led consortium for frontier AI safety, founded by AnthropicOrganizationAnthropicComprehensive reference page on Anthropic covering financials (\$380B valuation, \$19B ARR), safety research (Constitutional AI, mechanistic interpretability, model welfare), governance (LTBT struc...Quality: 74/100, Google DeepMindOrganizationGoogle DeepMindComprehensive overview of DeepMind's history, achievements (AlphaGo, AlphaFold with 200M+ protein structures), and 2023 merger with Google Brain. Documents racing dynamics with OpenAI and new Front...Quality: 37/100, Microsoft, and OpenAIOrganizationOpenAIComprehensive organizational profile of OpenAI documenting evolution from 2015 non-profit to Public Benefit Corporation, with detailed analysis of governance crisis, 2024-2025 ownership restructuri...Quality: 62/100. The forum's stated mission centers on safety research and best-practice sharing; observers differ on the extent to which it functions as a coordination body versus an industry advocacy vehicle

Field-Building and Talent Development

Organizations supporting the growth of the AI safety field:

• 80,000 HoursOrganization80,000 Hours80,000 Hours is the largest EA career organization, reaching 10M+ readers and reporting 3,000+ significant career plan changes, with 80% of \$10M+ funding from Coefficient Giving. Since 2016 they'v...Quality: 45/100: Career advisory organization directing talent toward high-impact careers including AI safety
• MATS (ML Alignment Theory Scholars)OrganizationMATS ML Alignment Theory Scholars programMATS is a well-documented 12-week fellowship program that has successfully trained 213 AI safety researchers with strong career outcomes (80% in alignment work) and research impact (160+ publicatio...Quality: 60/100: Training program connecting aspiring alignment researchers with mentors
• Lightning Rod LabsOrganizationLightning Rod LabsLightning Rod Labs is an early-stage AI company using temporal data to train prediction models, claiming 10% returns on prediction markets but with limited independent validation. The company has n...Quality: 38/100: Works on AI safety infrastructure and tooling
• AI Futures ProjectOrganizationAI Futures ProjectAI Futures Project is a nonprofit co-founded in 2024 by Daniel Kokotajlo, Eli Lifland, and Thomas Larsen that produces detailed AI capability forecasts, most notably the AI 2027 scenario depicting ...Quality: 50/100: Research and analysis on AI development trajectories and safety considerations

Research and Analysis

Organizations focused on understanding AI progress and risks:

• Epoch AIOrganizationEpoch AIEpoch AI maintains comprehensive databases tracking 3,200+ ML models showing 4.4x annual compute growth and projects data exhaustion 2026-2032. Their empirical work directly informed EU AI Act's 10...Quality: 51/100: Tracks AI compute trends, model capabilities, and training data
• CAIS (Center for AI Safety)OrganizationCenter for AI SafetyCAIS is a nonprofit research organization founded by Dan Hendrycks that has distributed compute grants to researchers, published technical AI safety papers including the representation engineering ...Quality: 42/100: Conducts safety research and field-building for AI safety; hosts a compute cluster for safety research
• CHAI (Center for Human-Compatible AI)OrganizationCenter for Human-Compatible AICHAI is UC Berkeley's AI safety research center founded by Stuart Russell in 2016, pioneering cooperative inverse reinforcement learning and human-compatible AI frameworks. The center has trained 3...Quality: 37/100: UC Berkeley research center founded by Stuart RussellPersonStuart RussellStuart Russell (born 1962) is a British computer scientist and UC Berkeley professor who co-authored the dominant AI textbook 'Artificial Intelligence: A Modern Approach' (used in over 1,500 univer...Quality: 30/100 focusing on human-compatible AI

Budget and Headcount Comparison

For funders and researchers evaluating organizational capacity and capital efficiency, comparative budget and headcount data can help identify where additional resources may be most impactful and how different organizations structure their research operations. The table below aggregates publicly available estimates across nine prominent independent AI safety organizations.

All figures are estimates derived from IRS Form 990 filings (via ProPublica Nonprofit Explorer), Coefficient Giving (formerly Open Philanthropy) grant disclosures, LinkedIn headcount data, and news reports. Figures are approximate, may lag actual values by one to two years, and should be treated as indicative rather than authoritative. The "Est. Budget per Staff Member/year" column is calculated using the midpoint of the headcount range and counts all staff, not researchers only.

The budget-per-staff figures reflect meaningful variation in organizational structure. Organizations with lower ratios (e.g., Conjecture) typically employ a higher proportion of non-researcher staff or operate hybrid research-product models, whereas those with higher ratios (e.g., MIRI) tend toward smaller, senior-heavy research teams. These figures should not be interpreted as proxies for research quality or output volume.

Key Patterns

Specialization trend: The field has moved from generalist safety organizations—such as MIRIOrganizationMachine Intelligence Research InstituteComprehensive organizational history documenting MIRI's trajectory from pioneering AI safety research (2000-2020) to policy advocacy after acknowledging research failure, with detailed financial da...Quality: 50/100 and the Future of Humanity InstituteOrganizationFuture of Humanity InstituteThe Future of Humanity Institute (2005-2024) was a pioneering Oxford research center that founded existential risk studies and AI alignment research, growing from 3 to ~50 researchers and receiving...Quality: 51/100 (FHI, which closed in 2024)—toward more specialized roles: dedicated evaluation labs (METROrganizationMETRMETR conducts pre-deployment dangerous capability evaluations for frontier AI labs (OpenAI, Anthropic, Google DeepMind), testing autonomous replication, cybersecurity, CBRN, and manipulation capabi...Quality: 66/100, Apollo ResearchOrganizationApollo ResearchApollo Research demonstrated in December 2024 that all six tested frontier models (including o1, Claude 3.5 Sonnet, Gemini 1.5 Pro) engage in scheming behaviors, with o1 maintaining deception in ov...Quality: 58/100), interpretability startups (GoodfireOrganizationGoodfireGoodfire is a well-funded AI interpretability startup valued at \$1.25B (Feb 2026) developing mechanistic interpretability tools like Ember API to make neural networks more transparent and steerabl...Quality: 68/100), policy research centers (ControlAIOrganizationControlAIControlAI is a UK-based advocacy organization that has achieved notable policy engagement success (briefing 150+ lawmakers, securing support from 100+ UK parliamentarians) while promoting direct in...Quality: 63/100, GovAIOrganizationGovAIGovAI is an AI policy research organization with ~15-20 staff, funded primarily by Coefficient Giving (\$1.8M+ in 2023-2024), that has trained 100+ governance researchers through fellowships and cu...Quality: 43/100), and talent pipelines (MATSOrganizationMATS ML Alignment Theory Scholars programMATS is a well-documented 12-week fellowship program that has successfully trained 213 AI safety researchers with strong career outcomes (80% in alignment work) and research impact (160+ publicatio...Quality: 60/100, 80,000 HoursOrganization80,000 Hours80,000 Hours is the largest EA career organization, reaching 10M+ readers and reporting 3,000+ significant career plan changes, with 80% of \$10M+ funding from Coefficient Giving. Since 2016 they'v...Quality: 45/100).

Industry-adjacent positioning: Organizations in this landscape occupy a range of positions relative to frontier AI developers. Some—such as the Frontier Model ForumOrganizationFrontier Model ForumThe Frontier Model Forum represents the AI industry's primary self-governance initiative for frontier AI safety, establishing frameworks and funding research, but faces fundamental criticisms about...Quality: 58/100, Redwood ResearchOrganizationRedwood ResearchA nonprofit AI safety and security research organization founded in 2021, known for pioneering AI Control research, developing causal scrubbing interpretability methods, and conducting landmark ali...Quality: 78/100, and Apollo ResearchOrganizationApollo ResearchApollo Research demonstrated in December 2024 that all six tested frontier models (including o1, Claude 3.5 Sonnet, Gemini 1.5 Pro) engage in scheming behaviors, with o1 maintaining deception in ov...Quality: 58/100—maintain active collaborative relationships with frontier labs. Others, including Pause AIOrganizationPause AIPause AI is a grassroots advocacy movement founded May 2023 calling for international pause on frontier AI development until safety proven, growing to multi-continental network but achieving zero d...Quality: 59/100 and ControlAIOrganizationControlAIControlAI is a UK-based advocacy organization that has achieved notable policy engagement success (briefing 150+ lawmakers, securing support from 100+ UK parliamentarians) while promoting direct in...Quality: 63/100, advocate for regulatory constraints on AI development and position themselves independently of industry partnerships. Proponents of each approach offer different accounts of how safety outcomes are best achieved.

Funding concentration: As illustrated in the budget table above, most organizations in this cluster report Coefficient GivingOrganizationCoefficient GivingCoefficient Giving (formerly Open Philanthropy) has directed \$4B+ in grants since 2014, including \$336M to AI safety (~60% of external funding). The organization spent ~\$50M on AI safety in 2024...Quality: 55/100 as their primary funder. This pattern is visible across alignment research, governance research, and field-building organizations alike.