Holden Karnofsky
Holden Karnofsky
Holden Karnofsky directed \$300M+ in AI safety funding through Coefficient Giving (formerly Open Philanthropy), growing the field from ~20 to 400+ FTE researchers and developing influential frameworks like the 'Most Important Century' thesis (15% transformative AI by 2036, 50% by 2060). His funding decisions include a \$580M Anthropic investment and establishment of 15+ university AI safety programs.
Holden Karnofsky
Holden Karnofsky directed \$300M+ in AI safety funding through Coefficient Giving (formerly Open Philanthropy), growing the field from ~20 to 400+ FTE researchers and developing influential frameworks like the 'Most Important Century' thesis (15% transformative AI by 2036, 50% by 2060). His funding decisions include a \$580M Anthropic investment and establishment of 15+ university AI safety programs.
Overview
Holden Karnofsky was co-CEO of Coefficient Giving↗🔗 webOpen Philanthropy grants databaseOpen Philanthropy provides grants across multiple domains including global health, catastrophic risks, and scientific progress. Their focus spans technological, humanitarian, an...x-riskresource-allocationresearch-prioritiesoptimization+1Source ↗ (formerly Coefficient Giving), the most influential grantmaker in AI safety and existential risk. Through Coefficient, he directed over $300 million toward AI safety research and governance, fundamentally transforming it from a fringe academic interest into a well-funded field with hundreds of researchers. In 2025, he joined Anthropic.
His strategic thinking has shaped how the effective altruism community prioritizes AI risk through frameworks like the "Most Important Century"↗🔗 web"Most Important Century"effective-altruismai-safety-fundingai-timelinesSource ↗ thesis. This argues we may live in the century that determines humanity's entire future trajectory due to transformative AI development.
| Funding Achievement | Amount | Impact |
|---|---|---|
| Total AI safety grants | $300M+ | Enabled field growth from ~dozens to hundreds of researchers |
| Anthropic investment | $580M+ | Created major safety-focused AI lab |
| Field building grants | $50M+ | Established academic programs and research infrastructure |
Risk Assessment
| Risk Category | Karnofsky's Assessment | Evidence | Timeline |
|---|---|---|---|
| Transformative AI | ~15% by 2036, ≈50% by 2060 | Bio anchors framework↗🔗 webBio anchors frameworkeffective-altruismai-safety-fundingai-timelinesSource ↗ | This century |
| Existential importance | "Most important century" | AI could permanently shape humanity's trajectory | 2021-2100 |
| Tractability | High enough for top priority | Open Phil's largest focus area allocation | Current |
| Funding adequacy | Severely underfunded | Still seeking to grow field substantially | Ongoing |
Career Evolution and Major Achievements
Early Career (2007-2014): Building Effective Altruism
| Period | Role | Key Achievements |
|---|---|---|
| 2007-2011 | Co-founder, GiveWell↗🔗 webGiveWellcost-effectivenessresearch-prioritiesexpected-valueeffective-altruism+1Source ↗ | Pioneered rigorous charity evaluation methodology |
| 2011-2014 | Launch Coefficient Giving | Expanded beyond global health to cause prioritization |
| 2012-2014 | EA movement building | Helped establish effective altruism as global movement |
Transition to AI Focus (2014-2018)
Initial AI engagement:
- 2014: First significant AI safety grants through Coefficient Giving (now Coefficient Giving)
- 2016: Major funding to Center for Human-Compatible AI (CHAI)
- 2017: Early OpenAI funding (before pivot to for-profit)
- 2018: Increased conviction leading to AI as top priority
AI Safety Leadership (2018-2025)
Major funding decisions:
- 2021: $580M investment in Anthropic↗🔗 web★★★★☆Anthropic$580M investment in Anthropiceffective-altruismai-safety-fundingai-timelinesSource ↗ to create safety-focused lab
- 2022: Establishment of AI safety university programs↗🔗 webAI safety university programssafetyeffective-altruismai-safety-fundingai-timelinesSource ↗
- 2023: Expanded governance funding addressing AI regulation
Departure from Coefficient Giving and Impact (2023-2025)
Karnofsky's departure from Coefficient Giving in 2023 had significant ripple effects on the organization's AI safety work. Ajeya Cotra, who worked closely with Karnofsky for nine years, described losing "an incredibly engaged partner... someone who would read 30 pages of analysis and give you deep feedback." His departure removed a key source of intellectual partnership that had driven Coefficient's AI strategy, including the Bio Anchors framework and the organization's approach to technical AI safety grantmaking. Karnofsky subsequently joined Anthropic, where he continues working on AI safety from within a frontier lab.
Strategic Frameworks and Intellectual Contributions
The "Most Important Century" Thesis
Core argument structure:
| Component | Claim | Implication |
|---|---|---|
| Technology potential | Transformative AI possible this century | Could exceed agricultural/industrial revolution impacts |
| Speed differential | AI transition faster than historical precedents | Less time to adapt and coordinate |
| Leverage moment | Our actions now shape outcomes | Unlike past revolutions where individuals had little influence |
| Conclusion | This century uniquely important | Justifies enormous current investment |
Supporting evidence:
- Biological anchors methodology↗🔗 webBio anchors frameworkeffective-altruismai-safety-fundingai-timelinesSource ↗ for AI timelines
- Historical analysis of technological transitions
- Economic modeling of AI impact potential
Bio Anchors Framework
Developed with Ajeya Cotra↗🔗 webAjeya Cotraeffective-altruismai-safety-fundingai-timelinesSource ↗, this framework estimates AI development timelines by comparing required computation to biological systems:
| Anchor Type | Computation Estimate | Timeline Implication |
|---|---|---|
| Human brain | ≈10^15 FLOP/s | Medium-term (2030s-2040s) |
| Human lifetime | ≈10^24 FLOP | Longer-term (2040s-2050s) |
| Evolution | ≈10^41 FLOP | Much longer-term if needed |
Coefficient Giving Funding Strategy
Portfolio Approach
| Research Area | Funding Focus | Key Recipients | Rationale |
|---|---|---|---|
| Technical alignment | $100M+ | Anthropic, Redwood Research | Direct work on making AI systems safer |
| AI governance | $80M+ | Center for Security and Emerging Technology↗🔗 web★★★★☆CSET GeorgetownCSET: AI Market DynamicsI apologize, but the provided content appears to be a fragmentary collection of references or headlines rather than a substantive document that can be comprehensively analyzed. ...prioritizationresource-allocationportfolioescalation+1Source ↗, policy fellowships | Institutional responses to AI development |
| Field building | $50M+ | University programs, individual researchers | Growing research community |
| Compute governance | $20M+ | Compute monitoring research | Oversight of AI development resources |
Grantmaking Philosophy
Key principles:
- Hits-based giving: Expect most grants to have limited impact, few to be transformative
- Long time horizons: Patient capital for 5-10 year research projects
- Active partnership: Strategic guidance beyond just funding
- Portfolio diversification: Multiple approaches given uncertainty
Notable funding decisions:
- Anthropic investment↗🔗 web★★★★☆Anthropic$580M investment in Anthropiceffective-altruismai-safety-fundingai-timelinesSource ↗: $580M to create safety-focused competitor to OpenAI
- MIRI funding: Early support for foundational AI alignment research
- Policy fellowships: Placing AI safety researchers in government positions
Current Views and Assessment
Karnofsky's AI Risk Timeline
Based on public statements and Coefficient Giving priorities from 2023-2024, Karnofsky's views reflect a combination of timeline estimates derived from technical forecasting and strategic assessments about field readiness and policy urgency:
| Expert/Source | Estimate | Reasoning |
|---|---|---|
| Transformative AI (2022) | 15% by 2036, 50% by 2060 | Derived from the bio anchors framework developed with Ajeya Cotra, which estimates AI development timelines by comparing required computation to biological systems. This central estimate suggests transformative AI is more likely than not within this century, though substantial uncertainty remains around both shorter and longer timelines. |
| Field adequacy (2024) | Still severely underfunded | Despite directing over $100M toward AI safety and growing the field from approximately 20 to 400+ FTE researchers, Coefficient Giving continues aggressive hiring and grantmaking. This assessment reflects the belief that the scale of the challenge—ensuring safe development of transformative AI—far exceeds current resources and talent devoted to it. |
| Policy urgency (2024) | High priority | Coefficient has significantly increased governance focus, funding policy research, placing fellows in government positions, and supporting regulatory frameworks. This shift recognizes that technical alignment work alone is insufficient—institutional and policy responses are critical to managing AI development trajectories and preventing racing dynamics. |
Evolution of Views (2020-2024)
| Year | Key Update | Reasoning |
|---|---|---|
| 2021 | "Most Important Century" series | Crystallized long-term strategic thinking |
| 2022 | Increased policy focus | Recognition of need for governance alongside technical work |
| 2023 | Anthropic model success | Validation of safety-focused lab approach |
| 2024 | Accelerated timelines concern | Shorter timelines than bio anchors suggested↗🔗 webShorter timelines than bio anchors suggestedeffective-altruismai-safety-fundingai-timelinesSource ↗ |
Influence on AI Safety Field
Field Growth Metrics
| Metric | 2015 | 2024 | Growth Factor |
|---|---|---|---|
| FTE researchers | ≈20 | ≈400 | 20x |
| Annual funding | <$5M | >$200M | 40x |
| University programs | 0 | 15+ | New category |
| Major organizations | 2-3 | 20+ | 7x |
Institutional Impact
Academic legitimacy:
- Funding enabled AI safety courses↗🔗 webAI safety coursessafetyeffective-altruismai-safety-fundingai-timelinesSource ↗ at major universities
- Supported tenure-track positions focused on alignment research
- Created pathway for traditional CS researchers to enter field
Policy influence:
- Funded experts now advising US AI Safety Institute
- Supported research informing EU AI Act↗🔗 webEU AI ActThe EU AI Act introduces the world's first comprehensive AI regulation, classifying AI applications into risk categories and establishing legal frameworks for AI development and...governancesoftware-engineeringcode-generationprogramming-ai+1Source ↗
- Built relationships between AI safety community and policymakers
Key Uncertainties and Strategic Cruxes
Open Questions in Karnofsky's Framework
| Uncertainty | Stakes | Current Evidence |
|---|---|---|
| AI timeline accuracy | Entire strategy timing | Mixed signals from recent capabilities |
| Technical tractability | Funding allocation efficiency | Early positive results but limited validation |
| Governance effectiveness | Policy investment value | Unclear institutional responsiveness |
| Anthropic success | Large investment justification | Strong early results but long-term unknown |
Strategic Disagreements
Within EA community:
- Some argue for longtermist focus beyond AI
- Others prefer global health and development↗🔗 webGiveWellcost-effectivenessresearch-prioritiesexpected-valueeffective-altruism+1Source ↗ emphasis
- Debate over concentration vs. diversification of funding
With AI safety researchers:
- Tension between technical alignment focus and governance approaches
- Disagreement over open vs. closed development funding
- Questions about emphasis on capabilities research safety benefits
Public Communication and Influence
Cold Takes Blog Impact
Most influential posts:
- "The Most Important Century"↗🔗 web"Most Important Century"effective-altruismai-safety-fundingai-timelinesSource ↗ series (>100k views)
- "AI Timelines: Where the Arguments Stand"↗🔗 webShorter timelines than bio anchors suggestedeffective-altruismai-safety-fundingai-timelinesSource ↗ (policy reference)
- "Bio Anchors" explanation↗🔗 webBio anchors frameworkeffective-altruismai-safety-fundingai-timelinesSource ↗ (research methodology)
Communication approach:
- Transparent reasoning and uncertainty acknowledgment
- Accessible explanations of complex topics
- Regular updates as views evolve
- Direct engagement with critics and alternative viewpoints
Media and Policy Engagement
| Platform | Reach | Impact |
|---|---|---|
| Congressional testimony | Direct policy influence | Informed AI regulation debate |
| Academic conferences | Research community | Shaped university AI safety programs |
| EA Global talks | Movement direction | Influenced thousands of career decisions |
| Podcast interviews | Public understanding | Mainstream exposure for AI safety ideas |
Current Priorities and Future Direction
2024-2026 Strategic Focus
Immediate priorities:
- Anthropic scaling: Supporting responsible development of powerful systems
- Governance acceleration: Policy research and implementation support
- Technical diversification: Funding multiple alignment research approaches
- International coordination: Supporting global AI safety cooperation
Emerging areas:
- Compute governance infrastructure
- AI evaluation methodologies
- Corporate AI safety practices
- Prediction market applications
Long-term Vision
Field development goals:
- Self-sustaining research ecosystem independent of Coefficient Giving
- Government funding matching or exceeding philanthropic support
- Integration of safety research into mainstream AI development
- International coordination mechanisms for AI governance
Critiques and Responses
Common Criticisms
| Criticism | Karnofsky's Response | Counter-evidence |
|---|---|---|
| Over-concentration of power | Funding diversification, transparency | Multiple other major funders emerging |
| Field capture risk | Portfolio approach, external evaluation | Continued criticism tolerated and addressed |
| Timeline overconfidence | Explicit uncertainty, range estimates | Regular updating based on new evidence |
| Governance skepticism | Measured expectations, multiple approaches | Early policy wins demonstrate tractability |
Ongoing Debates
Resource allocation:
- Should Coefficient Giving fund more basic research vs. applied safety work?
- Optimal balance between technical and governance approaches?
- Geographic distribution of funding (US-centric concerns)
Strategic approach:
- Speed vs. care in scaling funding
- Competition vs. cooperation with AI labs
- Public advocacy vs. behind-the-scenes influence
Sources & Resources
Primary Sources
| Type | Source | Description |
|---|---|---|
| Blog | Cold Takes↗🔗 webCold Takeseffective-altruismai-safety-fundingai-timelinesSource ↗ | Karnofsky's strategic thinking and analysis |
| Organization | Coefficient Giving↗🔗 webOpen Philanthropy grants databaseOpen Philanthropy provides grants across multiple domains including global health, catastrophic risks, and scientific progress. Their focus spans technological, humanitarian, an...x-riskresource-allocationresearch-prioritiesoptimization+1Source ↗ | Grant database and reasoning |
| Research | Bio Anchors Report↗🔗 webBio Anchors Reporteffective-altruismai-safety-fundingai-timelinesSource ↗ | Technical forecasting methodology |
| Testimony | Congressional Hearing↗🏛️ government★★★★★US CongressCongressional Hearingeffective-altruismai-safety-fundingai-timelinesSource ↗ | Policy positions and recommendations |
Secondary Analysis
| Type | Source | Focus |
|---|---|---|
| Academic | EA Research↗✏️ blog★★★☆☆EA ForumEA Forum Career Poststalentfield-buildingcareer-transitionsprioritization+1Source ↗ | Critical analysis of funding decisions |
| Journalistic | MIT Technology Review↗🔗 web★★★★☆MIT Technology ReviewMIT Technology Review: Deepfake Coverageai-forecastingcompute-trendstraining-datasetsconstitutional-ai+1Source ↗ | External perspective on influence |
| Policy | RAND Corporation↗🔗 web★★★★☆RAND CorporationRANDRAND conducts policy research analyzing AI's societal impacts, including potential psychological and national security risks. Their work focuses on understanding AI's complex im...governancecybersecurityprioritizationresource-allocation+1Source ↗ | Government research on philanthropic AI funding |
Related Profiles
- Dario Amodei - CEO of Anthropic, major funding recipient
- Paul Christiano - Technical alignment researcher, influenced Karnofsky's views
- Nick Bostrom - Author of "Superintelligence," early influence on Coefficient AI focus
- Eliezer Yudkowsky - MIRI founder, recipient of early Coefficient AI safety grants
References
Open Philanthropy provides grants across multiple domains including global health, catastrophic risks, and scientific progress. Their focus spans technological, humanitarian, and systemic challenges.
I apologize, but the provided content appears to be a fragmentary collection of references or headlines rather than a substantive document that can be comprehensively analyzed. Without a complete, coherent source text, I cannot generate a meaningful summary or review. To properly complete the task, I would need: 1. A full research document or article 2. Clear contextual content explaining the research's scope, methodology, findings 3. Sufficient detail to extract meaningful insights If you have the complete source document, please share it and I'll be happy to provide a thorough analysis following the specified JSON format. Would you like to: - Provide the full source document - Clarify the source material - Select a different document for analysis
The EU AI Act introduces the world's first comprehensive AI regulation, classifying AI applications into risk categories and establishing legal frameworks for AI development and deployment.