Holden Karnofsky
Page Status
Quality:40 (Adequate)⚠️
Importance:25 (Peripheral)
Last edited:2026-01-29 (3 days ago)
Words:1.7k
Backlinks:1
Structure:
📊 14📈 0🔗 41📚 0•30%Score: 10/15
LLM Summary:Holden Karnofsky directed $300M+ in AI safety funding through Open Philanthropy, growing the field from ~20 to 400+ FTE researchers and developing influential frameworks like the 'Most Important Century' thesis (15% transformative AI by 2036, 50% by 2060). His funding decisions include a $580M Anthropic investment and establishment of 15+ university AI safety programs.
Issues (1):
- QualityRated 40 but structure suggests 67 (underrated by 27 points)
See also:EA Forum
Researcher
Holden Karnofsky
Importance25
Websiteopenphilanthropy.org
RoleFormer Co-CEO (now at Anthropic)
Known ForDirecting billions toward AI safety, effective altruism leadership, AI timelines work
Overview
Section titled “Overview”Holden Karnofsky was co-CEO of Coefficient Giving↗🔗 webOpen Philanthropy grants databaseOpen Philanthropy provides grants across multiple domains including global health, catastrophic risks, and scientific progress. Their focus spans technological, humanitarian, an...Source ↗Notes (formerly Open Philanthropy), the most influential grantmaker in AI safety and existential risk. Through Coefficient, he directed over $100 million toward AI safety research and governance, fundamentally transforming it from a fringe academic interest into a well-funded field with hundreds of researchers. In 2025, he joined Anthropic.
His strategic thinking has shaped how the effective altruism community prioritizes AI risk through frameworks like the “Most Important Century”↗🔗 web"Most Important Century"Source ↗Notes thesis. This argues we may live in the century that determines humanity’s entire future trajectory due to transformative AI development.
| Funding Achievement | Amount | Impact |
|---|---|---|
| Total AI safety grants | $300M+ | Enabled field growth from ~dozens to hundreds of researchers |
| Anthropic investment | $580M+ | Created major safety-focused AI lab |
| Field building grants | $50M+ | Established academic programs and research infrastructure |
Risk Assessment
Section titled “Risk Assessment”| Risk Category | Karnofsky’s Assessment | Evidence | Timeline |
|---|---|---|---|
| Transformative AI | ~15% by 2036, ≈50% by 2060 | Bio anchors framework↗🔗 webBio anchors frameworkSource ↗Notes | This century |
| Existential importance | ”Most important century” | AI could permanently shape humanity’s trajectory | 2021-2100 |
| Tractability | High enough for top priority | Open Phil’s largest focus area allocation | Current |
| Funding adequacy | Severely underfunded | Still seeking to grow field substantially | Ongoing |
Career Evolution and Major Achievements
Section titled “Career Evolution and Major Achievements”Early Career (2007-2014): Building Effective Altruism
Section titled “Early Career (2007-2014): Building Effective Altruism”| Period | Role | Key Achievements |
|---|---|---|
| 2007-2011 | Co-founder, GiveWell↗🔗 webGiveWellSource ↗Notes | Pioneered rigorous charity evaluation methodology |
| 2011-2014 | Launch Coefficient Giving | Expanded beyond global health to cause prioritization |
| 2012-2014 | EA movement building | Helped establish effective altruism as global movement |
Transition to AI Focus (2014-2018)
Section titled “Transition to AI Focus (2014-2018)”Initial AI engagement:
- 2014: First significant AI safety grants through Coefficient (then Open Philanthropy)
- 2016: Major funding to Center for Human-Compatible AI (CHAI)Lab AcademicCHAICHAI is UC Berkeley's AI safety research center founded by Stuart Russell in 2016, pioneering cooperative inverse reinforcement learning and human-compatible AI frameworks. The center has trained 3...Quality: 37/100
- 2017: Early OpenAI funding (before pivot to for-profit)
- 2018: Increased conviction leading to AI as top priority
AI Safety Leadership (2018-Present)
Section titled “AI Safety Leadership (2018-Present)”Major funding decisions:
- 2021: $580M investment in Anthropic↗🔗 web★★★★☆Anthropic$580M investment in AnthropicSource ↗Notes to create safety-focused lab
- 2022: Establishment of AI safety university programs↗🔗 webAI safety university programsSource ↗Notes
- 2023: Expanded governance funding addressing AI regulation
Strategic Frameworks and Intellectual Contributions
Section titled “Strategic Frameworks and Intellectual Contributions”The “Most Important Century” Thesis
Section titled “The “Most Important Century” Thesis”Core argument structure:
| Component | Claim | Implication |
|---|---|---|
| Technology potential | Transformative AI possible this century | Could exceed agricultural/industrial revolution impacts |
| Speed differential | AI transition faster than historical precedents | Less time to adapt and coordinate |
| Leverage moment | Our actions now shape outcomes | Unlike past revolutions where individuals had little influence |
| Conclusion | This century uniquely important | Justifies enormous current investment |
Supporting evidence:
- Biological anchors methodology↗🔗 webBio anchors frameworkSource ↗Notes for AI timelines
- Historical analysis of technological transitions
- Economic modeling of AI impact potential
Bio Anchors Framework
Section titled “Bio Anchors Framework”Developed with Ajeya Cotra↗🔗 webAjeya CotraSource ↗Notes, this framework estimates AI development timelines by comparing required computation to biological systems:
| Anchor Type | Computation Estimate | Timeline Implication |
|---|---|---|
| Human brain | ≈10^15 FLOP/s | Medium-term (2030s-2040s) |
| Human lifetime | ≈10^24 FLOP | Longer-term (2040s-2050s) |
| Evolution | ≈10^41 FLOP | Much longer-term if needed |
Coefficient Giving Funding Strategy
Section titled “Coefficient Giving Funding Strategy”Portfolio Approach
Section titled “Portfolio Approach”| Research Area | Funding Focus | Key Recipients | Rationale |
|---|---|---|---|
| Technical alignment | $100M+ | AnthropicLabAnthropicComprehensive profile of Anthropic tracking its rapid commercial growth (from $1B to $7B annualized revenue in 2025, 42% enterprise coding market share) alongside safety research (Constitutional AI...Quality: 51/100, Redwood ResearchOrganizationRedwood ResearchRedwood Research is an AI safety lab founded in 2021 that has made significant contributions to mechanistic interpretability and, more recently, pioneered the "AI control" research agenda. | Direct work on making AI systems safer |
| AI governance | $80M+ | Center for Security and Emerging Technology↗🔗 web★★★★☆CSET GeorgetownCSET: AI Market DynamicsI apologize, but the provided content appears to be a fragmentary collection of references or headlines rather than a substantive document that can be comprehensively analyzed. ...Source ↗Notes, policy fellowships | Institutional responses to AI development |
| Field building | $50M+ | University programs, individual researchers | Growing research community |
| Compute governance | $20M+ | Compute monitoring researchMonitoringAnalyzes two compute monitoring approaches: cloud KYC (implementable in 1-2 years, covers ~60% of frontier training via AWS/Azure/Google) and hardware governance (3-5 year timeline). Cloud KYC targ...Quality: 69/100 | Oversight of AI development resources |
Grantmaking Philosophy
Section titled “Grantmaking Philosophy”Key principles:
- Hits-based giving: Expect most grants to have limited impact, few to be transformative
- Long time horizons: Patient capital for 5-10 year research projects
- Active partnership: Strategic guidance beyond just funding
- Portfolio diversification: Multiple approaches given uncertainty
Notable funding decisions:
- Anthropic investment↗🔗 web★★★★☆Anthropic$580M investment in AnthropicSource ↗Notes: $580M to create safety-focused competitor to OpenAI
- MIRI fundingOrganizationMIRIComprehensive organizational history documenting MIRI's trajectory from pioneering AI safety research (2000-2020) to policy advocacy after acknowledging research failure, with detailed financial da...Quality: 50/100: Early support for foundational AI alignment research
- Policy fellowships: Placing AI safety researchers in government positions
Current Views and Assessment
Section titled “Current Views and Assessment”Karnofsky’s AI Risk Timeline
Section titled “Karnofsky’s AI Risk Timeline”Based on public statements and Coefficient Giving priorities from 2023-2024, Karnofsky’s views reflect a combination of timeline estimates derived from technical forecasting and strategic assessments about field readiness and policy urgency:
| Expert/Source | Estimate | Reasoning |
|---|---|---|
| Transformative AI (2022) | 15% by 2036, 50% by 2060 | Derived from the bio anchors framework developed with Ajeya Cotra, which estimates AI development timelines by comparing required computation to biological systems. This central estimate suggests transformative AI is more likely than not within this century, though substantial uncertainty remains around both shorter and longer timelines. |
| Field adequacy (2024) | Still severely underfunded | Despite directing over $100M toward AI safety and growing the field from approximately 20 to 400+ FTE researchers, Coefficient Giving continues aggressive hiring and grantmaking. This assessment reflects the belief that the scale of the challenge—ensuring safe development of transformative AI—far exceeds current resources and talent devoted to it. |
| Policy urgency (2024) | High priority | Coefficient has significantly increased governance focus, funding policy research, placing fellows in government positions, and supporting regulatory frameworks. This shift recognizes that technical alignment work alone is insufficient—institutional and policy responses are critical to managing AI development trajectories and preventing racing dynamics. |
Evolution of Views (2020-2024)
Section titled “Evolution of Views (2020-2024)”| Year | Key Update | Reasoning |
|---|---|---|
| 2021 | ”Most Important Century” series | Crystallized long-term strategic thinking |
| 2022 | Increased policy focus | Recognition of need for governance alongside technical work |
| 2023 | Anthropic model success | Validation of safety-focused lab approach |
| 2024 | Accelerated timelines concern | Shorter timelines than bio anchors suggested↗🔗 webShorter timelines than bio anchors suggestedSource ↗Notes |
Influence on AI Safety Field
Section titled “Influence on AI Safety Field”Field Growth Metrics
Section titled “Field Growth Metrics”| Metric | 2015 | 2024 | Growth Factor |
|---|---|---|---|
| FTE researchers | ≈20 | ≈400 | 20x |
| Annual funding | <$5M | >$200M | 40x |
| University programs | 0 | 15+ | New category |
| Major organizations | 2-3 | 20+ | 7x |
Institutional Impact
Section titled “Institutional Impact”Academic legitimacy:
- Funding enabled AI safety courses↗🔗 webAI safety coursesSource ↗Notes at major universities
- Supported tenure-track positions focused on alignment research
- Created pathway for traditional CS researchers to enter field
Policy influence:
- Funded experts now advising US AI Safety InstituteOrganizationUS AI Safety InstituteThe US AI Safety Institute (AISI), established November 2023 within NIST with $10M budget (FY2025 request $82.7M), conducted pre-deployment evaluations of frontier models through MOUs with OpenAI a...Quality: 91/100
- Supported research informing EU AI Act↗🔗 webEU AI ActThe EU AI Act introduces the world's first comprehensive AI regulation, classifying AI applications into risk categories and establishing legal frameworks for AI development and...Source ↗Notes
- Built relationships between AI safety community and policymakers
Key Uncertainties and Strategic Cruxes
Section titled “Key Uncertainties and Strategic Cruxes”Open Questions in Karnofsky’s Framework
Section titled “Open Questions in Karnofsky’s Framework”| Uncertainty | Stakes | Current Evidence |
|---|---|---|
| AI timeline accuracy | Entire strategy timing | Mixed signals from recent capabilities |
| Technical tractability | Funding allocation efficiency | Early positive results but limited validation |
| Governance effectiveness | Policy investment value | Unclear institutional responsiveness |
| Anthropic success | Large investment justification | Strong early results but long-term unknown |
Strategic Disagreements
Section titled “Strategic Disagreements”Within EA community:
- Some argue for longtermist focus beyond AILong TimelinesComprehensive overview of the long-timelines worldview (20-40+ years to AGI, 5-20% P(doom)), arguing for foundational research over rushed solutions based on historical AI overoptimism, current sys...Quality: 91/100
- Others prefer global health and development↗🔗 webGiveWellSource ↗Notes emphasis
- Debate over concentration vs. diversification of funding
With AI safety researchers:
- Tension between technical alignment focusArgumentWhy Alignment Might Be HardComprehensive synthesis of why AI alignment is fundamentally difficult, covering specification problems (value complexity, Goodhart's Law), inner alignment failures (mesa-optimization, deceptive al...Quality: 61/100 and governance approaches
- Disagreement over open vs. closed developmentCruxOpen vs Closed Source AIComprehensive analysis of open vs closed source AI debate, documenting that open model performance gap narrowed from 8% to 1.7% in 2024, with 1.2B+ Llama downloads by April 2025 and DeepSeek R1 dem...Quality: 60/100 funding
- Questions about emphasis on capabilities research safety benefits
Public Communication and Influence
Section titled “Public Communication and Influence”Cold Takes Blog Impact
Section titled “Cold Takes Blog Impact”Most influential posts:
- “The Most Important Century”↗🔗 web"Most Important Century"Source ↗Notes series (>100k views)
- “AI Timelines: Where the Arguments Stand”↗🔗 webShorter timelines than bio anchors suggestedSource ↗Notes (policy reference)
- “Bio Anchors” explanation↗🔗 webBio anchors frameworkSource ↗Notes (research methodology)
Communication approach:
- Transparent reasoning and uncertainty acknowledgment
- Accessible explanations of complex topics
- Regular updates as views evolve
- Direct engagement with critics and alternative viewpoints
Media and Policy Engagement
Section titled “Media and Policy Engagement”| Platform | Reach | Impact |
|---|---|---|
| Congressional testimony | Direct policy influence | Informed AI regulation debateCruxGovernment Regulation vs Industry Self-GovernanceComprehensive comparison of government regulation versus industry self-governance for AI, documenting that US federal AI regulations doubled to 59 in 2024 while industry lobbying surged 141% to 648...Quality: 54/100 |
| Academic conferences | Research community | Shaped university AI safety programs |
| EA Global talks | Movement direction | Influenced thousands of career decisions |
| Podcast interviews | Public understanding | Mainstream exposure for AI safety ideas |
Current Priorities and Future Direction
Section titled “Current Priorities and Future Direction”2024-2026 Strategic Focus
Section titled “2024-2026 Strategic Focus”Immediate priorities:
- Anthropic scaling: Supporting responsible development of powerful systems
- Governance acceleration: Policy research and implementation support
- Technical diversification: Funding multiple alignment research approaches
- International coordination: Supporting global AI safety cooperation
Emerging areas:
- Compute governance infrastructure
- AI evaluationEvaluationComprehensive overview of AI evaluation methods spanning dangerous capability assessment, safety properties, and deception detection, with categorized frameworks from industry (Anthropic Constituti...Quality: 72/100 methodologies
- Corporate AI safetyCorporateMajor AI labs invest $300-500M annually in safety (5-10% of R&D) through responsible scaling policies and dedicated teams, but face 30-40% safety team turnover and significant implementation gaps b...Quality: 68/100 practices
- Prediction marketInterventionPrediction MarketsPrediction markets achieve Brier scores of 0.16-0.24 (15-25% better than polls) by aggregating dispersed information through financial incentives, with platforms handling $1-3B annually. For AI saf...Quality: 56/100 applications
Long-term Vision
Section titled “Long-term Vision”Field development goals:
- Self-sustaining research ecosystem independent of Coefficient Giving
- Government funding matching or exceeding philanthropic support
- Integration of safety research into mainstream AI development
- International coordination mechanisms for AI governance
Critiques and Responses
Section titled “Critiques and Responses”Common Criticisms
Section titled “Common Criticisms”| Criticism | Karnofsky’s Response | Counter-evidence |
|---|---|---|
| Over-concentration of power | Funding diversification, transparency | Multiple other major funders emerging |
| Field capture risk | Portfolio approach, external evaluation | Continued criticism tolerated and addressed |
| Timeline overconfidence | Explicit uncertainty, range estimates | Regular updating based on new evidence |
| Governance skepticism | Measured expectations, multiple approaches | Early policy wins demonstrate tractability |
Ongoing Debates
Section titled “Ongoing Debates”Resource allocation:
- Should Coefficient Giving fund more basic research vs. applied safety work?
- Optimal balance between technical and governance approaches?
- Geographic distribution of funding (US-centric concerns)
Strategic approach:
- Speed vs. care in scaling funding
- Competition vs. cooperation with AI labs
- Public advocacy vs. behind-the-scenes influence
Sources & Resources
Section titled “Sources & Resources”Primary Sources
Section titled “Primary Sources”| Type | Source | Description |
|---|---|---|
| Blog | Cold Takes↗🔗 webCold TakesSource ↗Notes | Karnofsky’s strategic thinking and analysis |
| Organization | Coefficient Giving↗🔗 webOpen Philanthropy grants databaseOpen Philanthropy provides grants across multiple domains including global health, catastrophic risks, and scientific progress. Their focus spans technological, humanitarian, an...Source ↗Notes | Grant database and reasoning |
| Research | Bio Anchors Report↗🔗 webBio Anchors ReportSource ↗Notes | Technical forecasting methodology |
| Testimony | Congressional Hearing↗🏛️ government★★★★★US CongressCongressional HearingSource ↗Notes | Policy positions and recommendations |
Secondary Analysis
Section titled “Secondary Analysis”| Type | Source | Focus |
|---|---|---|
| Academic | EA Research↗✏️ blog★★★☆☆EA ForumEA Forum Career PostsSource ↗Notes | Critical analysis of funding decisions |
| Journalistic | MIT Technology Review↗🔗 web★★★★☆MIT Technology ReviewMIT Technology Review: Deepfake CoverageSource ↗Notes | External perspective on influence |
| Policy | RAND Corporation↗🔗 web★★★★☆RAND CorporationRANDRAND conducts policy research analyzing AI's societal impacts, including potential psychological and national security risks. Their work focuses on understanding AI's complex im...Source ↗Notes | Government research on philanthropic AI funding |
Related Profiles
Section titled “Related Profiles”- Dario AmodeiResearcherDario AmodeiComprehensive biographical profile of Anthropic CEO Dario Amodei documenting his 'race to the top' philosophy, 10-25% catastrophic risk estimate, 2026-2030 AGI timeline, and Constitutional AI appro...Quality: 41/100 - CEO of Anthropic, major funding recipient
- Paul ChristianoResearcherPaul ChristianoComprehensive biography of Paul Christiano documenting his technical contributions (IDA, debate, scalable oversight), risk assessment (~10-20% P(doom), AGI 2030s-2040s), and evolution from higher o...Quality: 39/100 - Technical alignment researcher, influenced Karnofsky’s views
- Nick BostromResearcherNick BostromComprehensive biographical profile of Nick Bostrom covering his founding of FHI, the landmark 2014 book 'Superintelligence' that popularized AI existential risk, and key philosophical contributions...Quality: 25/100 - Author of “Superintelligence,” early influence on Coefficient AI focus
- Eliezer YudkowskyResearcherEliezer YudkowskyComprehensive biographical profile of Eliezer Yudkowsky covering his foundational contributions to AI safety (CEV, early problem formulation, agent foundations) and notably pessimistic views (>90% ...Quality: 35/100 - MIRI founder, recipient of early Coefficient AI safety grants
What links here
- Toby Ordresearcher