ControlAI

Quick Assessment

Key Links

Overview

ControlAI is a UK-based organization focused on AI safety and policy advocacy, with the mission to prevent the development of artificial superintelligence (ASI) and ensure humanity retains control over advanced AI systems.¹ The organization operates primarily through campaigns, policy proposals, and public engagement rather than technical research, emphasizing the need for democratic control over transformative AI development.

Founded in 2023 as an offshoot of Conjecture, ControlAI has positioned itself as one of the most professionalized AI activist groups, producing high-quality media campaigns and policy briefs targeted at lawmakers and the general public.² The organization's core tagline—"Fighting to keep humanity in control"—specifically targets control over deepfakesRiskDeepfakesComprehensive overview of deepfake risks documenting \$60M+ in fraud losses, 90%+ non-consensual imagery prevalence, and declining detection effectiveness (65% best accuracy). Reviews technical cap...Quality: 50/100, AI scaling, foundation models, and AI overall.³

ControlAI's primary theory of change centers on the "Direct Institutional Plan" (DIP), launched in March 2025, which promotes safe-by-design AI engineering, metrology of intelligence, and human-controlled transformative AI.⁴ The organization warns that no current methods exist to contain systems more intelligent than all humanity combined, echoing warnings from AI scientists, world leaders, and CEOs about potential human extinction risks.⁵

History

Founding and Early Development

ControlAI was founded in 2023 by Andrea Miotti, emerging as an offshoot of Conjecture, an AI startup led by Connor LeahyPersonConnor LeahyBiography of Connor Leahy, CEO and co-founder of Conjecture, an AI safety company based in London. Previously co-founded EleutherAI in 2020, which produced GPT-J and GPT-NeoX. Leahy holds a high P(...Quality: 19/100.⁶ The organization was established in the lead-up to the AI Safety Summit at Bletchley Park, UK, where it made a notable splash by hiring a blimp to fly over the summit as part of its advertising campaigns.⁷

Andrea Miotti, who holds a PhD in machine learning robustness and previously worked at Palantir and BCG, founded the organization after leading communications and policy efforts at Conjecture.⁸ The organization operates as a nonprofit "private company limited by guarantee" in the UK and as a 501(c)(4) nonprofit in the US.⁹

Evolution of Strategy

From its inception through 2024, ControlAI ran several major campaigns:

• October 2023: Prevented international endorsement of scaling policies at the AI Safety Summit¹⁰
• November-December 2023: Opposed exemptions for foundation models in the EU AI ActPolicyEU AI ActComprehensive overview of the EU AI Act's risk-based regulatory framework, particularly its two-tier approach to foundation models that distinguishes between standard and systemic risk AI systems. ...Quality: 55/100¹¹
• December 2023 - June 2024: Ran a major campaign against deepfakes¹²

The organization has evolved from a think tank model to focus on grassroots outreach and direct engagement with policymakers, transitioning to prevent ASI development via direct work.¹³

Recent Developments (2025-2026)

In March 2025, ControlAI launched "The Direct Institutional Plan" as their comprehensive strategy for achieving binding regulation on advanced AI systems.¹⁴ The UK pilot campaign, running from November 2024 through May 2025, demonstrated significant success: the organization briefed 84 cross-party UK parliamentarians (4 in 10 MPs, 3 in 10 Lords, and 2 in 10 from devolved legislatures), with over 20 publicly supporting their campaign for binding regulation.¹⁵

By December 2025, a huge coalition of lawmakers had called for binding regulation on powerful AI systems, representing a major milestone for the organization's advocacy efforts.¹⁶ As of early 2026, the organization continues scaling the DIP to the UK executive branch and expanding to the US and other countries.¹⁷

Leadership and Team

Andrea Miotti serves as the organization's public face, featured in media outlets and podcasts discussing AI extinction risks.¹⁸ The organization reportedly has 9 employees as of 2024.¹⁹

Policy Approach and Campaigns

The Direct Institutional Plan (DIP)

ControlAI's flagship initiative is the Direct Institutional Plan, a three-phase policy framework (Safety, Stability, Flourishing) that uses computing power as a proxy for AI capabilities.²⁰ The plan advocates for:

• Bans on superintelligence development: Prohibition of systems more intelligent than all humanity combined
• Dangerous capability restrictions: Preventing automated AI research, advanced hacking capabilities, and recursive self-improvement
• Pre-deployment demonstrations: Requiring developers to prove system safety before release
• AI development licensing: Establishing regulatory frameworks for advanced AI development
• Mandatory kill switches: Requiring emergency shutdown capabilities for advanced systems
• Compute cluster monitoring: Tracking large-scale AI training infrastructure²¹

The DIP is designed as a collaborative framework open to citizens and organizations worldwide, emphasizing independent participation rather than exclusive partnerships.²² ControlAI has developed country-specific policy briefs and offers advice to influential individuals and organizations via a dedicated partners page.²³

"A Narrow Path" Policy Framework

The organization's "A Narrow Path" policy paper underwent systematic evaluation through a policy sprint red-teamed by Apart Research in July 2025.²⁴ The sprint evaluated six policies with code released for reproducibility, demonstrating scalable monitoring of capability acquisition via phase transitions and agent dynamics simulations across eight sectors (from enterprise to critical infrastructure).²⁵

Advocacy Tools and Resources

ControlAI has created tools enabling citizens to contact lawmakers, executives, civil service, media, and civil society in their jurisdictions to advocate for superintelligence risk policies.²⁶ These tools have facilitated over 150,000 messages sent to lawmakers.²⁷

Impact and Achievements

Lawmaker Engagement

ControlAI's most significant achievement has been its success in engaging policymakers on AI extinction risks:

• Briefed 150+ lawmakers on AI extinction risk
• Secured support from 100+ UK parliamentarians for their campaign
• Achieved public endorsement from over 20 cross-party UK parliamentarians (more than 1 in 3 briefed)
• Drafted and presented 1 AI bill to the UK Prime Minister's office²⁸

The organization's cold-email campaign to British MPs and Lords engaged 70 representatives, with 31 publicly opposing ASI development—a remarkable conversion rate that defied initial predictions of resistance to strong extinction-risk messaging.²⁹

Public Opinion and Reach

ControlAI has commissioned multiple YouGov polls demonstrating strong UK public support for AI safety measures:

January 2025 YouGov Poll (UK):

• 73% favor halting rapid superintelligence development
• 74% support empowering the Artificial Intelligence Safety Institute (AISI) as regulator
• 87% support safety regime for AI development
• 76% favor monitoring large compute clusters
• 82% support mandatory AISI testing and company accountability³⁰

Additional Public Engagement:

• 79% support for a UK AI regulator
• 87% want developers to prove safety before release
• 150+ million views on AI risk content
• 150,000+ messages sent to lawmakers via their tools³¹

Media Presence

ControlAI has achieved significant media coverage, with mentions in:

• The Spectator (January 29, 2025) on DeepSeek stakes
• Newsweek (January 31, 2025) on AI extinction race
• Financial Times (September 12, 2024) on OpenAI bioweapon risks
• The Guardian (December 8, 2025) on parliamentarians calling for regulation
• City A.M. (December 8, 2025) on MPs pushing for stricter AI rules
• The Guardian (January 28, 2025) on former OpenAI researcher warnings
• New York Times (March 14, 2024) on powerful AI preparedness
• The Times (December 6, 2024) on schemingRiskSchemingScheming—strategic AI deception during training—has transitioned from theoretical concern to observed behavior across all major frontier models (o1: 37% alignment faking, Claude: 14% harmful compli...Quality: 74/100 ChatGPT³²

Special Projects

ControlAI has launched several targeted projects:

• "Artificial Guarantees" (January 2025): Documenting inconsistencies by AI companies, highlighting shifting statements on risks and broken promises³³
• "What leaders say about AI" (September 2024): Compilation of warnings from AI leaders and researchers³⁴
• Rational Animations collaboration: Video "What if AI just keeps getting smarter?" garnered 1.4 million views, warning of superintelligent, self-improving AI leading to extinction via indifference³⁵

Criticisms and Controversies

Critique of Coefficient Giving's Approach

ControlAI has positioned itself in opposition to Coefficient GivingOrganizationCoefficient GivingCoefficient Giving (formerly Open Philanthropy) has directed \$4B+ in grants since 2014, including \$336M to AI safety (~60% of external funding). The organization spent ~\$50M on AI safety in 2024...Quality: 55/100's approach to AI safety, arguing that the funder's strategy is "undemocratic" and centralizes control in a small group of "trusted" actors.³⁶ The organization's "Direct Institutional Plan" dedicates over 500 words to criticizing Coefficient Giving (now Coefficient GivingOrganizationCoefficient GivingCoefficient Giving (formerly Open Philanthropy) has directed \$4B+ in grants since 2014, including \$336M to AI safety (~60% of external funding). The organization spent ~\$50M on AI safety in 2024...Quality: 55/100) as the main funder in AI safety, highlighting:

• Over $80 million provided to establish the Center for Security and Emerging TechnologyOrganizationCSET (Center for Security and Emerging Technology)CSET is a \$100M+ Georgetown center with 50+ staff conducting data-driven AI policy research, particularly on U.S.-China competition and export controls. The center conducts hundreds of annual gove...Quality: 43/100 (CSET), which placed fellows in the US Department of Commerce and White House
• Funding for the Horizon Institute supporting placements in US congressional offices and executive agencies
• Grants to OpenAI in exchange for a board seat for Holden KarnofskyPersonHolden KarnofskyHolden Karnofsky directed \$300M+ in AI safety funding through Coefficient Giving (formerly Open Philanthropy), growing the field from ~20 to 400+ FTE researchers and developing influential framewo...Quality: 40/100
• Acting as "sole arbiter" of trustworthiness in AGI control strategy³⁷

ControlAI argues that Coefficient Giving's approach of building influence through strategic placements and supporting "responsible actors" building superintelligence (a view associated with figures like Holden KarnofskyPersonHolden KarnofskyHolden Karnofsky directed \$300M+ in AI safety funding through Coefficient Giving (formerly Open Philanthropy), growing the field from ~20 to 400+ FTE researchers and developing influential framewo...Quality: 40/100 and Will MacAskillPersonWill MacAskillComprehensive biographical reference on Will MacAskill covering his founding of EA organizations, academic work on moral uncertainty and longtermism, AGI preparedness advocacy, and controversies in...Quality: 60/100) is fundamentally flawed compared to their civic engagement model emphasizing democratic processes.³⁸

Industry Relationship Critiques

The organization has been vocal in criticizing frontier AI companies for what it characterizes as systematically undermining alignment research and regulation to race toward AGI. ControlAI argues that companies are driven by "utopian beliefs" in AGI ushering in an ideal world rather than prioritizing safety.³⁹

Specific criticisms include:

• Insufficient investment in alignment (only $200 million and a handful of researchers working on problems requiring decades and trillions)
• Companies collaborating to unlock resources like chips and power while ignoring governance
• Shifting baseline tactics and broken promises documented in their "Artificial Guarantees" project
• Racing to ASI despite warnings, downplaying risks even while acknowledging issues like bioweapon misuse⁴⁰

Skepticism About Plan Feasibility

Within the effective altruism and AI safety communities, ControlAI's approach has received mixed reception:

Positive Views:

• Described as the "most x-risk-focused" 501(c)(4) organization
• Praised for concrete campaigns with tangible results (31 public commitments from MPs/Lords)
• Collaboration with Rational Animations characterized as "really great"⁴¹

Criticisms:

• Donors and community members express skepticism that global "pause AIOrganizationPause AIPause AI is a grassroots advocacy movement founded May 2023 calling for international pause on frontier AI development until safety proven, growing to multi-continental network but achieving zero d...Quality: 59/100" regulations are feasible due to coordination challenges
• Concerns that detection without enforcement is insufficient—companies could ignore reports
• Debates over impact: videos effective for awareness but less successful at converting views to actions like emails or calls to action
• Tension with EA leadership favoring cautious superintelligence development over outright bans⁴²

Sensationalism Concerns

Some critics have characterized ControlAI as a group that "dramatically warns of AI's purported extinction risk," potentially sensationalizing risks.⁴³ However, CEO Andrea Miotti has responded that critics often nitpick experimental setups but should focus on trends in AI behaviors like self-preservation, resistance to shutdown, and strategic deception.⁴⁴

Relationship to AI Safety and Alignment

ControlAI operates primarily in the AI policy and advocacy space rather than technical alignment research. The organization's approach is grounded in the assessment that alignment is fundamentally intractable with current resources:

• Solving alignment would require decades of research and trillions in investment to address issues like identifying human values, reconciling contradictions, and predicting side-effects
• Currently only $200 million is invested, mostly in patching issues rather than solving core problems
• Progress is resource-limited rather than insight-limited, leading to opaque, rapidly advancing systems where experts fail to predict new skills or internal workings⁴⁵

The organization emphasizes that AI controlSafety AgendaAI ControlAI Control is a defensive safety approach that maintains control over potentially misaligned AI through monitoring, containment, and redundancy, offering 40-60% catastrophic risk reduction if align...Quality: 75/100 is not just a technical problem but requires institutional rules and democratic governance.⁴⁶ This positions ControlAI distinctly from technical alignment organizations like AnthropicOrganizationAnthropicComprehensive reference page on Anthropic covering financials (\$380B valuation, \$19B ARR), safety research (Constitutional AI, mechanistic interpretability, model welfare), governance (LTBT struc...Quality: 74/100, OpenAIOrganizationOpenAIComprehensive organizational profile of OpenAI documenting evolution from 2015 non-profit to Public Benefit Corporation, with detailed analysis of governance crisis, 2024-2025 ownership restructuri...Quality: 62/100's alignment teams, or Redwood ResearchOrganizationRedwood ResearchA nonprofit AI safety and security research organization founded in 2021, known for pioneering AI Control research, developing causal scrubbing interpretability methods, and conducting landmark ali...Quality: 78/100, which focus on developing technical solutions for controlling AI systems.

ControlAI's warnings align with broader concerns in the AI safety community about fundamental challenges in controlling superintelligent systems:

• Self-modifying code and learning from unanticipatable patterns make control potentially inherently insoluble
• Increasing AI capability reduces controllability; self-improving AI may resist goal changes and pursue instrumental goals like resource acquisition
• Verification is extremely difficult due to AI's software nature, enabling hiding of modifications⁴⁷

Community Reception

Effective Altruism and Rationalist Communities

Discussions on the EA Forum and LessWrongOrganizationLessWrongLessWrong is a rationality-focused community blog founded in 2009 that has influenced AI safety discourse, receiving \$5M+ in funding and serving as the origin point for ~31% of EA survey responden...Quality: 44/100 portray ControlAI as strongly x-risk-focused but reveal debates about the feasibility of their approach:

Support:

• Evolution from think tank to concrete grassroots campaigns praised
• High-quality content production acknowledged
• Donor support for their regulation efforts despite skepticism about global enforcement⁴⁸

Skepticism:

• Questions about whether moderate regulations or alignment research will succeed
• Concerns that weaker systems can't oversee stronger ones, with no known methods for superintelligent oversight
• Broader EA critiques that subhuman systems are inadequate for superintelligent oversight, requiring superhuman capability for proper alignment
• Unresolved issues around stability under reflection and steering stronger systems⁴⁹

The organization's positioning against prominent EA figures' views on "responsible actors" building superintelligence has created some tension with EA leadership.⁵⁰

Key Uncertainties

Several important questions remain about ControlAI's approach and impact:

1. Scalability of Success: Can the organization replicate its UK success in other countries, particularly the US where regulatory dynamics differ significantly?

2. Enforcement Mechanisms: How would proposed bans on superintelligence development be enforced internationally, given coordination challenges and incentive structures?

3. Technical Feasibility of Proposals: Are the organization's proposed capability thresholds and monitoring systems technically viable, and can they keep pace with rapid AI progress?

4. Relationship to Technical Safety Work: How does ControlAI's advocacy-first approach complement or conflict with technical alignment research efforts?

5. Long-term Funding Sustainability: With no major disclosed funders and only £1M expected in fundraising, can the organization sustain operations at the scale needed for global impact?

6. Impact on AI Development: Will the organization's campaigns lead to meaningful policy changes, or primarily serve to raise awareness without shifting development trajectories?

7. Alternative Approaches: Is preventing superintelligence development the optimal strategy, or should resources focus on alignment research, differential development, or other interventions?

Sources