Future of Humanity Institute (FHI)

📋Page Status

Page Type:ContentStyle Guide →Standard knowledge base article

Quality:51 (Adequate)⚠️

Importance:34.5 (Reference)

Last edited:2026-01-29 (3 days ago)

Words:4.2k

Backlinks:1

Structure:

📊 32📈 2🔗 3📚 24•5%Score: 14/15

LLM Summary:The Future of Humanity Institute (2005-2024) was a pioneering Oxford research center that founded existential risk studies and AI alignment research, growing from 3 to ~50 researchers and receiving $10M+ in funding before closing due to administrative conflicts. FHI produced seminal works (Superintelligence, The Precipice), trained leaders now at Anthropic/DeepMind/GovAI, and advised UN/UK government, demonstrating both transformative intellectual impact and the challenges of housing speculative research in traditional academia.

Issues (2):

QualityRated 51 but structure suggests 93 (underrated by 42 points)
Links5 links could use <R> components

Quick Assessment

Dimension	Assessment	Evidence
Focus	Existential Risk Research	AI safety, global catastrophic risks, macrostrategy, human enhancement
Status	Closed (April 16, 2024)	Faculty of Philosophy ended contracts; staff dispersed
Peak Size	≈50 researchers	Grew from 3 in 2005 to peak around 2018-2020
Duration	19 years (2005-2024)	Founded as 3-year pilot, became permanent institution
Total Funding	$10M+ from Coefficient Giving	Plus £1M from Elon Musk, ERC grants, Leverhulme Trust
Key Publications	Superintelligence (2014), The Precipice (2020)	Both became international bestsellers
Policy Reach	UN, UK Government, EU	Advised UN Secretary General, quoted by UK PM at UN
Spin-offs	GovAI, influenced CSER, GPI	Multiple organizations founded by alumni

Organization Details

Attribute	Details
Full Name	Future of Humanity Institute
Type	University Research Institute
Founded	November 2005
Closed	April 16, 2024
Location	University of Oxford, Faculty of Philosophy
Institutional Home	Oxford Martin School (initially James Martin 21st Century School)
Founder & Director	Nick Bostrom
Peak Staff	≈50 researchers
Website	fhi.ox.ac.uk (archived)
Final Report	FHI Final Report (Sandberg, 2024)
Major Funders	Coefficient Giving ($10M+), Elon Musk (£1M), ERC, Leverhulme Trust

Overview

The Future of Humanity Institute (FHI) was a multidisciplinary research center at the University of Oxford that fundamentally shaped how humanity thinks about long-term risks and the future of civilization. Founded by philosopher Nick Bostrom in November 2005 as part of the Oxford Martin School (then the James Martin 21st Century School), FHI brought together researchers from philosophy, computer science, mathematics, and economics to tackle questions that most of academia considered too speculative or far-fetched to study rigorously.

During its 19-year existence, FHI achieved an extraordinary record of intellectual impact relative to its modest size. The institute was involved in the germination of a wide range of ideas that have since become mainstream concerns: existential risk, effective altruism, longtermism, AI alignment, AI governance, global catastrophic risk, information hazards, the unilateralist’s curse, and moral uncertainty. Starting with just three researchers in 2005, FHI grew to approximately fifty at its peak before administrative conflicts with Oxford’s Faculty of Philosophy led to a hiring freeze in 2020 and ultimate closure in April 2024.

FHI’s influence extends far beyond its publications. The institute trained a generation of researchers who now hold leadership positions at Anthropic, DeepMind, OpenAI, the Centre for the Governance of AI (GovAI), and numerous other organizations. Toby Ord’s The Precipice was quoted by UK Prime Minister Boris Johnson in his 2021 UN General Assembly address, and FHI researchers advised the UN Secretary General’s Office on existential risk and future generations. The institute’s closure represents the end of an era, but its intellectual legacy continues through its alumni, spin-off organizations, and the fields it created.

Historical Evolution

Loading diagram...

Founding Era (2005-2008)

Nick Bostrom established FHI in November 2005 after recognizing that questions about humanity’s long-term future and existential risks were being systematically neglected by mainstream academia. The institute was initially funded as a three-year pilot project but quickly demonstrated its value through a series of influential publications and conferences.

Milestone	Date	Significance
FHI Founded	November 2005	First academic institute dedicated to existential risk
Initial Team	2005	3 researchers: Bostrom, plus initial hires
Oxford Martin School Integration	2005	Provided institutional legitimacy and infrastructure
Global Catastrophic Risks (book)	2008	First comprehensive academic treatment of GCR

In its early years, FHI focused on establishing existential risk as a legitimate field of academic inquiry. Bostrom’s 2002 paper “Existential Risk: Analyzing Human Extinction Scenarios” had laid the conceptual groundwork, defining existential risk as “one where an adverse outcome would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential.” FHI’s task was to build an institutional home for this research.

Growth Period (2008-2014)

Between 2008 and 2010, FHI hosted the Global Catastrophic Risks conference, wrote 22 academic journal articles, and published 34 chapters in academic volumes. The GCR conference was a pivotal moment in building an academic community around reducing risks to humanity’s future.

Achievement	Year	Impact
GCR Conference Series	2008-2010	Built academic community around catastrophic risk
22 Journal Articles	2008-2010	Established academic legitimacy
34 Book Chapters	2008-2010	Spread ideas across disciplines
Bostrom begins Superintelligence	2009	Originally one chapter on AI, grew into landmark book
Superintelligence published	July 2014	International bestseller; ignited AI safety movement

This period saw FHI expand its research scope significantly. When Bostrom began work on a book about existential risk in 2009, he found one chapter on AI “getting out of hand.” The issue of risks from superintelligent systems turned out to be much deeper than initially expected, eventually evolving into Superintelligence: Paths, Dangers, Strategies, published in 2014.

Peak Period (2014-2020)

The publication of Superintelligence marked the beginning of FHI’s most influential period. The book became an international bestseller and is credited with convincing many technologists, including Elon Musk and Bill Gates, to take AI risks seriously. FHI grew to approximately 50 researchers and received its largest funding commitments.

Development	Year	Details
Superintelligence impact	2014+	Read by Musk, Gates; influenced industry leaders
Elon Musk donation	2015	£1M for AI safety research
Governance of AI team formed	2018	Led by Allan Dafoe; later spins out
£13.3M Coefficient Giving grant	2018	Largest grant in Faculty of Philosophy history
The Precipice published	March 2020	First book-length treatment of existential risk for popular audience

In 2018, FHI received a series of awards totaling up to £13.3 million over three years from Coefficient Giving (then Open Philanthropy), the largest donation in the history of the Faculty of Philosophy at Oxford. This funding supported work on risks from advanced AI, biosecurity and pandemic preparedness, and macrostrategy.

Decline and Closure (2020-2024)

Despite its intellectual success, FHI’s final years were marked by what Anders Sandberg called “gradual suffocation by Faculty bureaucracy.” The flexible, fast-moving approach of the institute did not function well with the rigid rules and slow decision-making of the surrounding organization.

Event	Date	Consequence
Faculty hiring freeze begins	2020	No new researchers could be hired
Faculty fundraising freeze	2020	Couldn’t pursue new grants
GovAI spins out	2021	Largest team leaves to escape restrictions
Bostrom email controversy	2023	1996 email resurfaces; Oxford investigates
Faculty announces no contract renewals	Late 2023	Remaining staff told contracts would end
FHI officially closes	April 16, 2024	19-year run ends
Bostrom resigns from Oxford	2024	Founds Macrostrategy Research Initiative

The stated reason for closure, according to Bostrom, was that the university did not have the operational bandwidth to manage FHI. Sandberg explained the cultural mismatch: “I often described Oxford like a coral reef of calcified institutions built on top of each other… FHI was one such fish but grew too big for its hole. At that point it became either vulnerable to predators, or had to enlarge the hole, upsetting the neighbors.”

Research Programs and Contributions

Existential Risk Studies

FHI essentially created the academic field of existential risk studies. Before FHI, the topic was considered too speculative for serious academic attention. FHI demonstrated that it was possible to do rigorous research on big-picture questions about humanity’s future.

Research Area	Key Contributions	Impact
Definition and Taxonomy	Bostrom’s existential risk framework	Standard definitions used across field
Probability Estimation	Upper bounds on background extinction rate	Quantified risks for policy discussions
Fermi Paradox	”Dissolving the Fermi Paradox” (2018)	Showed we may be alone in observable universe
Vulnerable World Hypothesis	Bostrom (2019)	Framework for technology governance

The 2018 paper “Dissolving the Fermi Paradox” by Sandberg, Drexler, and Ord was the first to estimate and rigorously account for uncertainties in each term of the Drake equation. They found a high likelihood that we are alone in our galaxy or even the entire observable universe.

AI Safety Research

FHI was one of the earliest academic institutions to take AI safety seriously, working in close collaboration with labs such as DeepMind, OpenAI, and CHAI (Center for Human-Compatible AI).

Publication	Authors	Year	Contribution
”Racing to the precipice”	Armstrong, Shulman, Bostrom	2013	Modeled AI development race dynamics
Superintelligence	Bostrom	2014	Comprehensive analysis of superintelligence risks
”Safely interruptible agents”	Orseau, Armstrong	2016	Technical AI safety contribution
”Reframing Superintelligence”	Drexler	2019	Alternative “CAIS” model of AI development
”Truthful AI”	Evans et al.	2021	Framework for developing AI that doesn’t lie

Stuart Armstrong’s collaboration with DeepMind on “Interruptibility” was mentioned in over 100 media articles and represented one of FHI’s more practical AI safety contributions.

AI Governance

FHI’s Governance of AI (GovAI) team, led by Allan Dafoe, became the largest research group focused on the policy implications of advanced AI before spinning out as an independent organization in 2021.

Publication	Authors	Year	Focus
”AI Governance: A Research Agenda”	Dafoe	2018	Foundational governance framework
”The Malicious Use of AI”	Brundage et al.	2018	Security implications of AI
”Strategic implications of openness”	Bostrom	2017	Open vs. closed AI development

GovAI spun out of Oxford in 2021 specifically to “escape bureaucratic restrictions” and has since become an independent nonprofit. Allan Dafoe now heads DeepMind’s Long-Term AI Strategy and Governance Team.

Macrostrategy Research

FHI’s Macrostrategy group examined how long-term outcomes for humanity are connected to present-day actions—a research program that influenced the effective altruism movement’s focus on cause prioritization.

Concept	Originator	Significance
Information Hazards	Bostrom	Framework for managing dangerous knowledge
Unilateralist’s Curse	Bostrom	Explains why groups make riskier decisions
Moral Uncertainty	MacAskill, Ord	How to act under ethical uncertainty
Crucial Considerations	Bostrom	Factors that could reverse strategic priorities

Biosecurity Research

FHI’s Biosecurity group worked on making the world more secure against both natural and human-made catastrophic biological risks, anticipating many concerns that became urgent during the COVID-19 pandemic.

Key Personnel

Nick Bostrom (Founder and Director)

Attribute	Details
Role	Founder, Director (2005-2024)
Background	PhD Philosophy (LSE, 2000); BA Physics, Philosophy (Stockholm)
Key Works	Anthropic Bias (2002), Superintelligence (2014), Deep Utopia (2024)
Famous For	Simulation argument, existential risk framework, superintelligence analysis
Current Role	Principal Researcher, Macrostrategy Research Initiative

Bostrom is best known for his work in five areas: existential risk, the simulation argument, anthropics, impacts of future technology, and implications of consequentialism for global strategy. His simulation argument posits that one of three propositions must be true: civilizations almost never reach technological maturity, technologically mature civilizations are uninterested in running simulations, or we are almost certainly living in a simulation.

Toby Ord (Senior Research Fellow)

Attribute	Details
Role	Senior Research Fellow
Background	Computer Science turned Philosophy
Key Work	The Precipice: Existential Risk and the Future of Humanity (2020)
Co-founded	Giving What We Can (pledged to give most of earnings to charity)
Current Role	AI Governance, Oxford Martin School
Policy Impact	Advised UN Secretary General; quoted by UK PM at UN

Ord’s The Precipice was the first book-length treatment of existential risk for a wide audience, influencing policy in the United Kingdom and at the United Nations. Multiple FHI staff were invited to present their work to the British Parliament.

Anders Sandberg (Senior Research Fellow)

Attribute	Details
Role	Senior Research Fellow
Background	Neuroscience, Computational Neuroscience
Research Focus	Human enhancement, whole brain emulation, grand futures
Key Papers	”Dissolving the Fermi Paradox” (2018), FHI Final Report (2024)
Upcoming Work	Grand Futures (mapping physical limits of advanced civilizations)
Current Role	Mimir Center for Long Term Futures Research

Sandberg authored the FHI Final Report, which provides a detailed account of the institute’s history and achievements. He is described as a futurist who explored the outer limits of what advanced civilizations might achieve.

Stuart Armstrong (Research Fellow)

Attribute	Details
Role	Research Fellow
Research Focus	AI Safety, Value Alignment, Corrigibility
Key Publications	Smarter Than Us (2014), “Safely Interruptible Agents” (2016)
DeepMind Collaboration	”Interruptibility” paper mentioned in 100+ media articles
Current Role	Co-founder, AI safety startup

Armstrong’s research centered on how to define AI goals and map humanity’s partially-defined values into AI systems. His collaboration with DeepMind on interruptibility was one of FHI’s most visible practical contributions.

Eric Drexler (Senior Research Fellow)

Attribute	Details
Role	Senior Research Fellow
Background	Pioneer of nanotechnology; MIT PhD
Key Works	Engines of Creation (1986), “Reframing Superintelligence” (2019)
FHI Contribution	CAIS (Comprehensive AI Services) framework
Current Role	Senior Research Fellow, RAND Europe

Drexler, best known for pioneering the concept of molecular nanotechnology, brought a unique perspective to AI safety. His “Reframing Superintelligence” proposed an alternative to the “single superintelligent agent” model that dominated much AI safety thinking.

Carl Shulman (Research Associate)

Attribute	Details
Role	Research Associate
Research Focus	AI forecasting, AI impacts, embryo selection
Key Papers	”Racing to the precipice” (2013), “Embryo selection for cognitive enhancement” (2014)
Collaborations	Multiple papers with Bostrom on AI development dynamics

Shulman contributed to FHI’s work on forecasting AI development timelines and understanding the strategic implications of advanced AI.

Other Notable Researchers

Researcher	Focus Area	Current Position
Allan Dafoe	AI Governance	DeepMind Long-Term AI Strategy
Owain Evans	AI value learning	Academic researcher
Robin Hanson	Economics, Prediction Markets	George Mason University
Miles Brundage	AI Policy	Policy researcher

Major Publications

Books

Title	Author(s)	Year	Impact
Global Catastrophic Risks	Bostrom, Cirkovic (eds.)	2008	First comprehensive academic treatment
Human Enhancement	Savulescu, Bostrom (eds.)	2009	Bioethics of human augmentation
Superintelligence: Paths, Dangers, Strategies	Bostrom	2014	International bestseller; sparked AI safety movement
Smarter Than Us	Armstrong	2014	Accessible introduction to AI alignment
The Precipice: Existential Risk and the Future of Humanity	Ord	2020	Policy-influential treatment of existential risk
Moral Uncertainty	MacAskill, Bykvist, Ord	2020	Philosophical foundations of EA
Deep Utopia: Life and Meaning in a Solved World	Bostrom	2024	Post-scarcity philosophy

Influential Papers

Paper	Authors	Year	Citations	Key Contribution
”Existential Risks: Analyzing Human Extinction Scenarios”	Bostrom	2002	1000+	Founded the field
”The Superintelligent Will”	Bostrom	2012	High	Instrumental convergence thesis
”Thinking Inside the Box: Oracle AI”	Armstrong, Sandberg, Bostrom	2012	Moderate	AI containment strategies
”Racing to the Precipice”	Armstrong, Shulman, Bostrom	2013	Moderate	AI race dynamics
”Future Progress in AI: Expert Survey”	Muller, Bostrom	2016	High	First systematic AI timeline survey
”Dissolving the Fermi Paradox”	Sandberg, Drexler, Ord	2018	High	Rigorous Drake equation analysis
”The Vulnerable World Hypothesis”	Bostrom	2019	High	Technology governance framework
”Reframing Superintelligence: CAIS”	Drexler	2019	Moderate	Alternative AI development model

Funding and Resources

Coefficient Giving Grants

Grant	Amount	Year	Purpose
General Support	$1,995,425	2016	Unrestricted reserves, junior staff
Research Scholars Programme	$1,586,224	Various	Future scholars hiring
Major Grant	$12,250,810	2018	AI, biosecurity, macrostrategy (£13.3M total)
DPhil Positions	$139,263	Various	Doctoral student support
Admin/Operations	$100,000	Various	Via Effective Ventures

Total Coefficient Giving funding exceeded $10 million over FHI’s lifetime.

Other Major Funders

Funder	Amount	Year	Focus
Elon Musk (via FLI)	£1,000,000	2015	AI safety research
European Research Council	Various	Multiple	Research grants
Leverhulme Trust	Various	Multiple	Research grants
Survival and Flourishing Fund	≈$150,000	Various	General support

Budget and Operations

FHI’s annual revenues and expenses were approximately £1 million per year at operational scale, with the bulk of funding from academic grants that were “lumpy and hard to predict.”

Policy Impact

United Nations

Activity	Details
Secretary General Advisory	Toby Ord advised on existential risk and future generations
Human Development Report 2020	FHI contributed analysis
Boris Johnson UN Speech 2021	Quoted Toby Ord’s The Precipice

United Kingdom

Activity	Details
Parliamentary Presentations	Multiple staff invited to present to Parliament
Future Proof Report 2021	Co-authored UK resilience strategy report
Paymaster General Speech	Favorably mentioned FHI’s resilience work

European Union

FHI researchers contributed to policy discussions on AI governance that informed the development of the EU AI Act.

Loading diagram...

Direct Spin-offs

Organization	Founded	Connection	Current Status
Centre for the Governance of AI (GovAI)	2018 (spun out 2021)	FHI’s largest team	Independent nonprofit
Giving What We Can	2009	Co-founded by Toby Ord	Part of Effective Ventures
Macrostrategy Research Initiative	2024	Founded by Bostrom post-FHI	Active nonprofit
Mimir Center for Long Term Futures Research	2024	Anders Sandberg’s new home	New research center

Organization	Relationship	Focus
Global Priorities Institute	Shared staff, similar mission	EA-aligned research on prioritization
Oxford Martin School	FHI’s institutional home	Hosts multiple future-focused centers
Centre for Effective Altruism	Shared office space historically	EA movement hub

Organizations Influenced by FHI

Organization	Influence Type
Centre for the Study of Existential Risk (Cambridge)	FHI provided intellectual model
Anthropic	Multiple FHI alumni
DeepMind Safety Team	FHI collaborations, alumni
Future of Life Institute	Shared funders, mission alignment
MIRI	Intellectual exchange, some shared funders

Reasons for Closure

Official Reasons

The University stated it did not have “operational bandwidth” to manage FHI. The institute cited “increasing administrative headwinds within the Faculty of Philosophy.”

Detailed Analysis

Factor	Details	Impact
Hiring Freeze (2020)	Faculty prohibited new hires	Lost ability to replace departing researchers
Fundraising Freeze (2020)	Faculty prohibited new grant applications	Couldn’t pursue growth opportunities
Cultural Mismatch	Flexible startup style vs. rigid academia	Constant friction over procedures
Administrative Burden	Faculty bureaucracy increased over time	”Gradual suffocation” per Sandberg
Contract Non-Renewal (2023)	Faculty decided not to renew remaining contracts	Made closure inevitable

Anders Sandberg’s Explanation

In the FHI Final Report, Sandberg explained:

“While FHI had achieved significant academic and policy impact, the final years were affected by a gradual suffocation by Faculty bureaucracy. The flexible, fast-moving approach of the institute did not function well with the rigid rules and slow decision-making of the surrounding organization.”

He used the metaphor of Oxford as a “coral reef of calcified institutions built on top of each other,” with FHI as a fish that grew too big for its hole.

Contextual Factors

Factor	Timing	Potential Impact
Bostrom email controversy	2023	1996 racist email resurfaced; Oxford investigated
EA/FTX crisis	2022	Broader scrutiny of EA-affiliated organizations
Post-pandemic environment	2020+	University administrative changes

The university explicitly stated that the email controversy was not a factor in the closure decision, according to Bostrom.

Legacy and Assessment

Intellectual Legacy

Contribution	Significance	Current Status
Existential Risk Studies	Created the academic field	Now studied at multiple universities
AI Safety Research	Pioneered academic study	Major focus at top AI labs
AI Governance	Founded the subfield	GovAI and others continue work
Longtermism	Developed philosophical framework	Central to effective altruism
Information Hazards	Created conceptual framework	Standard consideration in biosecurity

Institutional Legacy

FHI demonstrated that it was possible to do rigorous academic research on big-picture questions about humanity’s future. Topics that once “struggled to eke out a precarious existence at the margins of a single philosophy department are now pursued by leading AI labs, government agencies, nonprofits, and specialized academic research centers.”

Alumni Impact

Destination	Notable Alumni
DeepMind	Allan Dafoe (AI Governance), others
AI Safety Startups	Stuart Armstrong (co-founder)
Oxford Martin School	Toby Ord (AI Governance)
Mimir Center	Anders Sandberg
Macrostrategy Research Initiative	Nick Bostrom
RAND Europe	Eric Drexler

Assessment of Impact

Dimension	Assessment	Evidence
Academic Influence	Transformative	Created multiple fields; thousands of citations
Policy Influence	Significant	UN, UK government engagement
Field Building	Exceptional	Trained generation of researchers
Organizational Model	Partially Failed	Administrative conflicts ended the institute
Timing	Good	Existed during critical period for AI safety awareness

Lessons and Implications

For Research Institutes

Lesson	Context	Implication
Institutional Fit Matters	FHI’s flexibility clashed with Oxford bureaucracy	Consider organizational culture carefully
Success Can Create Problems	Growth strained administrative relationships	Plan for scaling challenges
Spin-outs as Strategy	GovAI escaped by becoming independent	Independence may be worth pursuing early

For the AI Safety Field

FHI’s closure coincides with AI safety becoming mainstream. As Bostrom noted, “There is now a much broader support base for the kind of work FHI was set up to enable, so the institute essentially served its purpose.” The question is whether the distributed ecosystem of organizations can match FHI’s record of fundamental advances.

Open Questions

Did FHI’s administrative troubles reflect fixable problems or inherent tensions between academic institutions and existential risk research?
Will the distributed ecosystem of FHI successor organizations be as productive as the concentrated institute?
What institutional models best support long-term, speculative research?