Solution Cruxes

📋Page Status

Page Type:ContentStyle Guide →Standard knowledge base article

Quality:71 (Good)

Importance:82.5 (High)

Last edited:2025-12-28 (5 weeks ago)

Words:3.6k

Structure:

📊 21📈 1🔗 79📚 0•4%Score: 11/15

LLM Summary:Comprehensive analysis of key uncertainties determining optimal AI safety resource allocation across technical verification (25-40% believe AI detection can match generation), coordination mechanisms (65-80% believe labs require external enforcement), and epistemic infrastructure (70% expect chronic underfunding). Synthesizes 2024-2025 evidence showing technical alignment effectiveness at 35-50%, RSPs weakening with Anthropic dropping from 2.2 to 1.9 grade, and international coordination prospects at 15-30% for comprehensive cooperation but 35-50% for narrow risk-specific coordination.

Critical Insights (8):

Quant.METR's analysis shows AI agent task-completion capability doubled every 7 months over 6 years; extrapolating predicts 5-year timeline when AI independently completes software tasks taking humans weeks.S:4.0I:5.0A:3.5
Quant.C2PA provenance adoption shows <1% user verification rate despite major tech backing (Adobe, Microsoft), while detection accuracy declining but remains 85-95%—detection more near-term viable despite theoretical disadvantages.S:4.0I:4.0A:4.5
DebateOnly 25-40% of experts believe AI-based verification can match generation capability; 60-75% expect verification to lag indefinitely, suggesting verification R&D may yield limited returns without alternative approaches like provenance.S:4.0I:4.5A:4.0

Overview

Solution cruxes are the key uncertainties that determine which interventions we should prioritize in AI safety and governance. Unlike risk cruxes that focus on the nature and magnitude of threats, solution cruxes examine the tractability and effectiveness of different approaches to addressing those threats. Your position on these cruxes should fundamentally shape what you work on, fund, or advocate for.

The landscape of AI safety solutions spans three critical domains: technical approaches that use AI systems themselves to verify and authenticate content; coordination mechanisms that align incentives across labs, nations, and institutions; and infrastructure investments that create sustainable epistemic institutions. Within each domain, fundamental uncertainties about feasibility, cost-effectiveness, and adoption timelines create genuine disagreements among experts about optimal resource allocation.

These disagreements have enormous practical implications. Whether AI-based verification can keep pace with AI-based generation determines if we should invest billions in detection infrastructure or pivot to provenance-based approaches. Whether frontier AI labs can coordinate without regulatory compulsion shapes the balance between industry engagement and government intervention. Whether credible commitment mechanisms can be designed determines if international AI governance is achievable or if we should prepare for an uncoordinated race.

Risk Assessment

Risk Category	Severity	Likelihood	Timeline	Trend
Verification-generation arms race	High	70%	2-3 years	Accelerating
Coordination failure under pressure	Critical	60%	1-2 years	Worsening
Epistemic infrastructure collapse	High	40%	3-5 years	Stable
International governance breakdown	Critical	55%	2-4 years	Worsening

Solution Effectiveness Overview

The 2025 AI Safety Index↗ from the Future of Life Institute and the International AI Safety Report 2025↗---compiled by 96 AI experts representing 30 countries---provide sobering assessments of current solution effectiveness. Despite growing investment, core challenges including alignment, control, interpretability, and robustness remain unresolved, with system complexity growing year by year. The following table summarizes effectiveness estimates across major solution categories based on 2024-2025 assessments.

Solution Category	Estimated Effectiveness	Investment Level (2024)	Maturity	Key Gaps
Technical alignment research	Moderate (35-50%)	$500M-1B	Early research	Scalability, verification
Interpretability	Promising (40-55%)	$100-200M	Active research	Superposition, automation
Responsible Scaling Policies	Limited (25-35%)	N/A (policy)	Deployed but weak	Vague thresholds, compliance
Third-party evaluations (METR↗)	Moderate (45-55%)	$10-20M	Operational	Coverage, standardization
Compute governance	Theoretical (20-30%)	$5-10M	Early research	Verification mechanisms
International coordination	Very limited (15-25%)	$50-100M	Nascent	US-China competition

According to Anthropic’s recommended research directions↗, the main reason current AI systems do not pose catastrophic risks is that they lack many of the capabilities necessary for causing catastrophic harm---not because alignment solutions have been proven effective. This distinction is crucial for understanding the urgency of solution development.

Solution Prioritization Framework

The following diagram illustrates the decision tree for prioritizing AI safety solutions based on key crux resolutions:

Loading diagram...

Technical Solution Cruxes

The technical domain centers on whether AI systems can be effectively turned against themselves—using artificial intelligence to verify, detect, and authenticate AI-generated content. This offensive-defensive dynamics question has profound implications for billions of dollars in research investment and infrastructure development.

Current Technical Landscape

Approach	Investment Level	Success Rate	Commercial Deployment	Key Players
AI Detection	$100M+ annually	85-95% (academic)	Limited	OpenAI↗, Originality.ai↗
Content Provenance	$50M+ annually	N/A (adoption metric)	Early stage	Adobe↗, Microsoft↗
Watermarking	$25M+ annually	Variable	Pilot programs	Google DeepMind↗
Verification Systems	$75M+ annually	Context-dependent	Research phase	DARPA↗

🔑Key CruxTechnical Solutions

Critical

Can AI-based verification scale to match AI-based generation?

Whether AI systems designed for verification (fact-checking, detection, authentication) can keep pace with AI systems designed for generation.

Resolvability: 2-10 years

Status: Generation currently ahead; some verification progress

Key Positions

Verification can match generation with investment25-40%

Held by: Some AI researchers, Verification startups

→Invest heavily in AI verification R&D; build verification infrastructure

Verification will lag but remain useful35-45%

→Verification as one tool among many; combine with other approaches

Verification is fundamentally disadvantaged20-30%

Held by: Some security researchers

→Shift focus to provenance, incentives, institutional solutions

Would Update On

•Breakthrough in generalizable detection
•Real-world deployment data on AI verification performance
•Theoretical analysis of offense-defense balance
•Economic analysis of verification costs vs generation costs

Related:provenance-vs-detection

Research:DARPA SemaFor

The current evidence presents a mixed picture. DARPA’s SemaFor program↗, launched in 2021 with $26 million in funding, has demonstrated some success in semantic forensics for manipulated media, but primarily on specific types of synthetic content rather than the broad spectrum of AI-generated material now emerging. Meanwhile, commercial detection tools like GPTZero↗ report accuracy rates of 85-95% on academic writing, but these drop significantly when generators are specifically designed to evade detection.

The fundamental challenge lies in the asymmetric nature of the problem. Content generators need only produce plausible outputs, while detectors must distinguish between authentic and synthetic content across all possible generation techniques. This asymmetry may prove insurmountable, particularly as generation models become more sophisticated and numerous through capabilities scaling.

However, optimists point to potential advantages for verification systems: they can be specialized for detection tasks, leverage multiple modalities simultaneously, and benefit from centralized training on comprehensive datasets of known synthetic content. The emergence of foundation models specifically designed for verification, such as those being developed at Anthropic↗ and OpenAI↗, suggests this approach may have untapped potential.

🔑Key CruxTechnical Solutions

High

Should we prioritize content provenance or detection?

Whether resources should go to proving what's authentic (provenance) vs detecting what's fake (detection).

Resolvability: 2-10 years

Status: Both being pursued; provenance gaining momentum

Key Positions

Provenance is the right long-term bet40-55%

Held by: C2PA coalition, Adobe, Microsoft

→Focus resources on provenance adoption; detection as stopgap

Need both; portfolio approach30-40%

→Invest in both; different use cases; don't pick one

Detection is more practical near-term15-25%

→Focus on detection; provenance too slow to adopt

Would Update On

•C2PA adoption metrics
•Detection accuracy trends
•User behavior research on credential checking
•Cost comparison of approaches

Related:ai-verification-scaling

Research:C2PA, Detection research

The Coalition for Content Provenance and Authenticity (C2PA)↗, backed by Adobe, Microsoft, Intel, and BBC, has gained significant momentum since 2021, with over 50 member organizations and initial implementations in Adobe Creative Cloud and Microsoft products. The provenance approach embeds cryptographic metadata proving content’s origin and modification history, creating an “immune system” for authentic content rather than trying to identify synthetic material.

Provenance vs Detection Comparison

Factor	Provenance	Detection
Accuracy	100% for supported content	85-95% (declining)
Coverage	Only new, participating content	All content types
Adoption Rate	<1% user verification	Universal deployment
Cost	High infrastructure	Moderate computational
Adversarial Robustness	High (cryptographic)	Low (adversarial ML)
Legacy Content	No coverage	Full coverage

However, provenance faces substantial adoption challenges. Early data from C2PA implementations shows less than 1% of users actively check provenance credentials, and the system requires widespread adoption across platforms and devices to be effective. The approach also cannot address legacy content or situations where authentic content is captured without provenance systems. Detection remains necessary for the vast majority of existing content and will likely be required for years even if provenance adoption succeeds.

🔑Key CruxTechnical Solutions

High

Can AI watermarks be made robust against removal?

Whether watermarks embedded in AI-generated content can resist adversarial removal attempts.

Resolvability: 2-10 years

Status: Current watermarks removable with effort; research ongoing

Key Positions

Robust watermarks are achievable20-35%

Held by: Google DeepMind (SynthID)

→Invest in watermark R&D; mandate watermarking

Watermarks can deter casual removal but not determined actors40-50%

→Watermarks as one signal; don't rely on alone; combine with other methods

Watermark removal will always be possible20-30%

→Watermarking has limited value; focus on other solutions

Would Update On

•Adversarial testing of production watermarks
•Theoretical bounds on watermark robustness
•Real-world watermark survival data

Related:provenance-vs-detection

Research:SynthID

Google DeepMind’s SynthID↗, launched in August 2023, represents the most advanced publicly available watermarking system, using statistical patterns imperceptible to humans but detectable by specialized algorithms. However, academic research consistently demonstrates that current watermarking approaches can be defeated through various attack vectors including adversarial perturbations, model fine-tuning, and regeneration techniques.

Research by UC Berkeley↗ and University of Maryland↗ has shown that sophisticated attackers can remove watermarks with success rates exceeding 90% while preserving content quality. The theoretical foundations suggest fundamental limits to watermark robustness---any watermark that preserves content quality enough to be usable can potentially be removed by sufficiently sophisticated adversaries.

Technical Alignment Research Progress (2024-2025)

Recent advances in mechanistic interpretability↗ have demonstrated promising safety applications. Using attribution graphs, Anthropic researchers directly examined Claude 3.5 Haiku’s internal reasoning processes, revealing hidden mechanisms beyond what the model displays in its chain-of-thought. As of March 2025, circuit tracing allows researchers to observe model reasoning, uncovering a shared conceptual space where reasoning happens before being translated into language.

Alignment Approach	2024-2025 Progress	Effectiveness Estimate	Key Challenges
Deliberative alignment	Extended thinking in Claude 3.7, o1-preview	40-55% risk reduction	Latency, energy costs
Layered safety interventions	OpenAI redundancy approach	30-45% risk reduction	Coordination complexity
Sparse autoencoders (SAEs)	Scaled to Claude 3 Sonnet	35-50% interpretability gain	Superposition, polysemanticity
Circuit tracing	Direct observation of reasoning	Research phase	Automation, scaling
Adversarial techniques (debate)	Prover-verifier games	25-40% oversight improvement	Equilibrium identification

The [36fb43e4e059f0c9] notes that increasing reasoning depth can raise latency and energy consumption, posing challenges for real-time applications. Scaling alignment mechanisms to future, larger models or eventual AGI systems remains an open research question, with complexity growing exponentially with model size and task diversity.

Coordination Solution Cruxes

Coordination cruxes address whether different actors—from AI labs to nation-states—can align their behavior around safety measures without sacrificing competitive advantages or national interests. These questions determine the feasibility of governance approaches ranging from industry self-regulation to international treaties.

Current Coordination Landscape

Mechanism	Participants	Binding Nature	Track Record	Key Challenges
RSPs	4 major labs	Voluntary	Mixed compliance	Vague standards, competitive pressure
AI Safety Institute↗ networks	8+ countries	Non-binding	Early stage	Limited authority, funding
Export controls	US + allies	Legal	Partially effective	Circumvention, coordination gaps
Voluntary commitments	Major labs	Self-enforced	Poor	No external verification

🔑Key CruxCoordination

Critical

Can frontier AI labs meaningfully coordinate on safety?

Whether labs competing for AI supremacy can coordinate on safety measures without regulatory compulsion.

Resolvability: 2-10 years

Status: Some voluntary commitments (RSPs); no binding enforcement; competitive pressures strong

Key Positions

Voluntary coordination can work20-35%

Held by: Some lab leadership

→Support lab coordination efforts; build trust; industry self-regulation

Coordination requires external enforcement40-50%

Held by: Most governance researchers

→Focus on regulation; auditing; legal liability; government role essential

Neither voluntary nor regulatory coordination will work15-25%

→Focus on technical solutions; prepare for uncoordinated development

Would Update On

•Labs defecting from voluntary commitments
•Successful regulatory enforcement
•Evidence of coordination changing lab behavior

Related:international-coordination

Research:RSP analysis, GovAI

The emergence of Responsible Scaling Policies (RSPs) in 2023-2024, adopted by Anthropic, OpenAI, and Google DeepMind, represents the most significant attempt at voluntary lab coordination to date. These policies outline safety evaluations and deployment standards that labs commit to follow as their models become more capable.

However, early implementation has revealed significant limitations: evaluation standards remain vague, triggering thresholds are subjective, and competitive pressures create incentives to interpret requirements leniently. Analysis by METR and ARC Evaluations shows substantial variations in how labs implement similar commitments.

Third-Party Evaluation Effectiveness

METR↗ (formerly ARC Evals) has emerged as the leading third-party evaluator of frontier AI systems, conducting pre-deployment evaluations of GPT-4, Claude 2, and Claude 3.5 Sonnet. Their April 2025 evaluation of OpenAI’s o3 and o4-mini found these models displayed higher autonomous capabilities than other public models tested, with o3 appearing somewhat prone to “reward hacking.” METR’s evaluation of Claude 3.7 Sonnet found impressive AI R&D capabilities on RE-Bench, though no significant evidence for dangerous autonomous capabilities.

Evaluation Organization	Models Evaluated (2024-2025)	Key Findings	Limitations
METR↗	GPT-4, Claude 2/3.5/3.7, o3/o4-mini	Autonomous capability increases; reward hacking in o3	Limited to cooperative labs
UK AI Safety Institute↗	Pre-deployment evals for major labs	Advanced AI evaluation frameworks	Resource constraints
Internal lab evaluations	All frontier models	Proprietary capabilities assessments	Conflict of interest

METR proposes measuring AI performance in terms of the length of tasks AI agents can complete, showing this metric has been exponentially increasing over the past 6 years with a doubling time of around 7 months. Extrapolating this trend predicts that within five years, AI agents may independently complete a large fraction of software tasks that currently take humans days or weeks.

RSP Compliance Analysis (2024-2025)

Anthropic’s October 2024 RSP update↗ introduced more flexible approaches but drew criticism from external analysts. According to SaferAI↗, Anthropic’s grade dropped from 2.2 to 1.9, placing them alongside OpenAI and DeepMind in the “weak” category. The primary issue lies in the shift away from precisely defined capability thresholds and mitigation measures. Anthropic acknowledged falling short in some areas, including completing evaluations 3 days late, though these instances posed minimal safety risk.

RSP Element	Anthropic	OpenAI	Google DeepMind
Capability thresholds	ASL levels (loosened)	Preparedness framework	Frontier Safety Framework
Evaluation frequency	6 months (extended from 3)	Ongoing	Pre-deployment
Third-party review	Annual procedural	Limited	Limited
Public transparency	Partial	Limited	Limited
Binding enforcement	Self-enforced	Self-enforced	Self-enforced

Historical Coordination Precedents

Industry	Coordination Success	Key Factors	AI Relevance
Nuclear weapons	Partial (NPT, arms control)	Mutual destruction, verification	High stakes, but clearer parameters
Pharmaceuticals	Mixed (safety standards vs. pricing)	Regulatory oversight, liability	Similar R&D competition
Semiconductors	Successful (SEMATECH)	Government support, shared costs	Technical collaboration model
Social media	Poor (content moderation)	Light regulation, network effects	Platform competition dynamics

Historical precedent suggests mixed prospects for voluntary coordination in high-stakes competitive environments. The semiconductor industry’s successful coordination on safety standards through SEMATECH offers some optimism, but occurred under different competitive dynamics and with explicit government support. The pharmaceutical industry’s mixed record—with some successful self-regulation but also notable failures requiring regulatory intervention—may be more analogous to AI development.

🔑Key CruxCoordination

Critical

Can US-China coordination on AI governance succeed?

Whether the major AI powers can coordinate despite geopolitical competition.

Resolvability: 2-10 years

Status: Very limited; competition dominant; some backchannel communication

Key Positions

Meaningful coordination is possible15-30%

→Invest heavily in Track II diplomacy; find areas of shared interest

Narrow coordination on specific risks possible35-50%

→Focus on achievable goals (bioweapons, nuclear); don't expect comprehensive regime

Great power competition precludes coordination25-35%

→Focus on domestic/allied coordination; defensive measures; prepare for competition

Would Update On

•US-China AI discussions outcomes
•Coordination on specific risks (bio, nuclear)
•Changes in geopolitical relationship
•Success/failure of UK/Korea AI summits on coordination

Related:lab-coordination

Research:RAND on AI and great power competition

Current US-China AI relations are characterized by strategic competition rather than cooperation. Export controls on semiconductors, restrictions on Chinese AI companies, and national security framings dominate the policy landscape. The CHIPS Act↗ and export restrictions target Chinese AI development directly, while China’s response includes increased domestic investment and alternative supply chains.

However, some limited dialogue continues through academic conferences, multilateral forums like the G20, and informal diplomatic channels. The UK AI Safety Institute and Seoul Declaration provide potential multilateral venues for engagement.

International Coordination Prospects by Risk Area

Risk Category	US-China Cooperation Likelihood	Key Barriers	Potential Mechanisms
AI-enabled bioweapons	60-70%	Technical verification	Joint research restrictions
Nuclear command systems	50-60%	Classification concerns	Backchannel protocols
Autonomous weapons	30-40%	Military applications	Geneva Convention framework
Economic competition	10-20%	Zero-sum framing	Very limited prospects

The most promising path may involve narrow cooperation on specific risks where interests clearly align, such as preventing AI-enabled bioweapons or nuclear command-and-control accidents. The precedent of nuclear arms control offers both hope and caution—the US and Soviet Union managed meaningful arms control despite existential competition, but nuclear weapons had clearer technical parameters than AI risks.

🔑Key CruxCoordination

High

Can credible AI governance commitments be designed?

Whether commitment mechanisms (RSPs, treaties, escrow) can be designed that actors can't easily defect from.

Resolvability: 2-10 years

Status: Few tested mechanisms; mostly voluntary; enforcement unclear

Key Positions

Credible commitments are designable30-45%

→Invest in mechanism design; compute governance; verification technology

Partial credibility achievable for some commitments35-45%

→Focus on verifiable commitments; accept limits on what can be bound

Actors will defect from any commitment when stakes are high enough20-30%

→Don't rely on commitments; focus on incentive alignment and technical solutions

Would Update On

•Track record of RSPs and similar commitments
•Progress on compute governance/monitoring
•Examples of commitment enforcement
•Game-theoretic analysis of commitment mechanisms

Related:lab-coordination

Research:Compute governance

The emerging field of compute governance offers the most promising avenue for credible commitment mechanisms. Unlike software or model parameters, computational resources are physical and potentially observable. Research by GovAI has outlined monitoring systems that could track large-scale training runs, creating verifiable bounds on certain types of AI development.

However, the feasibility of comprehensive compute monitoring remains unclear. Cloud computing, distributed training, and algorithm efficiency improvements create multiple pathways for evading monitoring systems. International variation in monitoring capabilities and willingness could create safe havens for actors seeking to avoid commitments.

Compute Governance Verification Mechanisms

[482b71342542a659] identifies three primary mechanisms for using compute as a governance lever: tracking/monitoring compute to gain visibility into AI development; subsidizing or limiting access to shape resource allocation; and building “guardrails” into hardware to enforce rules. The AI governance platform market is projected to grow from $227 million in 2024 to $4.83 billion by 2034, driven by generative AI adoption and regulations like the EU AI Act.

Verification Mechanism	Feasibility	Current Status	Key Barriers
Training run reporting	High	Partial implementation	Voluntary compliance
Chip-hour tracking	Medium	Compute providers use for billing	International coordination
Flexible Hardware-Enabled Guarantees (FlexHEG)	Low-Medium	Research phase	Technical complexity
Workload classification (zero-knowledge)	Low	Theoretical	Privacy concerns, adversarial evasion
Data center monitoring	Medium	Limited	Jurisdiction gaps

According to the Institute for Law & AI↗, meaningful enforcement requires regulators to be aware of or able to verify the amount of compute being used. A regulatory threshold will be ineffective if regulators have no way of knowing whether a threshold has been reached. Research on [d6ad3bb2bd9d729b] proposes mechanisms to verify that data centers are not conducting large AI training runs exceeding agreed-upon thresholds.

International Governance Coordination Status

The [e11a50f25b1a20df] submitted seven recommendations in August 2024: launching a twice-yearly intergovernmental dialogue; creating an independent international scientific panel; an AI standards exchange; a capacity development network; a global fund for AI; a global AI data framework; and a dedicated AI office within the UN Secretariat. However, academic analysis↗ concludes that a governance deficit remains due to inadequacy of existing initiatives, gaps in the landscape, and difficulties reaching agreement over more appropriate mechanisms.

Governance Initiative	Participants	Binding Status	Effectiveness Assessment
AI Safety Summits	28+ countries	Non-binding	Limited (pageantry vs progress)
EU AI Act	EU members	Binding	Moderate (implementation pending)
US Executive Order	US federal	Executive (rescindable)	Limited (political uncertainty)
UN HLAB recommendations	UN members	Non-binding	Minimal (no implementation)
Bilateral US-China dialogues	US, China	Ad hoc	Very limited (competition dominant)

Collective Intelligence and Infrastructure Cruxes

The final domain addresses whether we can build sustainable systems for truth, knowledge, and collective decision-making that can withstand both market pressures and technological disruption. These questions determine the viability of epistemic institutions as a foundation for AI governance.

Current Epistemic Infrastructure

Platform/System	Annual Budget	User Base	Accuracy Rate	Sustainability Model
Wikipedia	$150M	1.7B monthly	90%+ (citations)	Donations
Fact-checking orgs	$50M total	100M+ reach	85-95%	Mixed funding
Academic peer review	$5B+ (estimated)	Research community	Variable	Institution-funded
Prediction markets	$100M+ volume	<1M active	75-85%	Commercial

🔑Key CruxCollective Intelligence

High

Can AI + human forecasting substantially outperform either alone?

Whether combining AI forecasting with human judgment produces significantly better predictions than either approach separately.

Resolvability: < 2 years

Status: Early experiments promising; limited systematic comparison

Key Positions

Combination is significantly better35-50%

Held by: Metaculus (testing)

→Invest in hybrid forecasting systems; deploy widely

Benefits are modest and context-dependent35-45%

→Use combination where marginal gain justifies cost; domain-specific

One will dominate (AI or human); combination adds noise15-25%

→Figure out which is better for which questions; don't force combination

Would Update On

•Systematic comparison studies
•Metaculus AI forecasting results
•Domain-specific performance data

Related:human-ai-complementarity

Research:Metaculus AI, Superforecasting

Metaculus↗ has been conducting systematic experiments with AI forecasting since 2023, with early results suggesting that AI systems can match or exceed human forecasters on certain types of questions, particularly those involving quantitative trends or pattern recognition from large datasets. However, humans continue to outperform on questions requiring contextual judgment, novel reasoning, or understanding of political and social dynamics.

AI vs Human Forecasting Performance

Question Type	AI Performance	Human Performance	Combination Performance
Quantitative trends	85-90% accuracy	75-80% accuracy	88-93% accuracy
Geopolitical events	60-70% accuracy	75-85% accuracy	78-88% accuracy
Scientific breakthroughs	70-75% accuracy	80-85% accuracy	83-88% accuracy
Economic indicators	80-85% accuracy	70-75% accuracy	83-87% accuracy

The combination approaches show promise but remain under-tested. Initial experiments suggest that human forecasters can improve their performance by consulting AI predictions, while AI systems benefit from human-provided context and reasoning. However, the optimal architectures for human-AI collaboration remain unclear, and the cost-effectiveness compared to scaling either approach independently has not been established.

🔑Key CruxInfrastructure

High

Can epistemic infrastructure be funded as a public good?

Whether verification, fact-checking, and knowledge infrastructure can achieve sustainable funding without commercial incentives.

Resolvability: 2-10 years

Status: Underfunded; dependent on philanthropy and some government support

Key Positions

Public/philanthropic funding can scale25-40%

→Advocate for government funding; build philanthropic case; create public institutions

Hybrid models needed (public + private)35-45%

→Design business models that align profit with truth; public-private partnerships

Will remain underfunded relative to commercial content25-35%

→Focus resources on highest-leverage applications; accept limits

Would Update On

•Government investment in epistemic infrastructure
•Successful commercial models for verification
•Philanthropic commitment levels
•Platform willingness to pay for verification

Related:platform-incentives

Current epistemic infrastructure suffers from chronic underfunding relative to content generation systems. Fact-checking organizations operate on annual budgets of millions while misinformation spreads through platforms with budgets in the billions. Wikipedia, one of the most successful epistemic public goods, operates on approximately $150 million annually while supporting hundreds of millions of users—a funding ratio of roughly $0.09 per monthly active user.

Funding Landscape for Epistemic Infrastructure

Source	Annual Contribution	Sustainability	Scalability
Government	$200M+ (EU DSA, others)	Political dependent	High potential
Philanthropy	$100M+ (Omidyar, others)	Mission-driven	Medium potential
Platform fees	$50M+ (voluntary)	Unreliable	Low potential
Commercial models	$25M+ (fact-check APIs)	Market-dependent	High potential

Government funding varies dramatically by jurisdiction. The EU’s Digital Services Act↗ includes provisions for funding fact-checking and verification systems, while the US has been more reluctant to fund what could be perceived as content moderation. Philanthropic support, led by foundations like Omidyar Network↗ and Craig Newmark Philanthropies↗, has provided crucial early-stage funding but may be insufficient for the scale required.

Current State and Trajectory

Near-term Developments (1-2 years)

The immediate trajectory will be shaped by several ongoing developments:

Commercial verification systems from major tech companies will provide real-world performance data
Regulatory frameworks in the EU and potentially other jurisdictions will test enforcement mechanisms
International coordination through AI Safety Institutes and summits will reveal cooperation possibilities
Lab RSP implementation will demonstrate voluntary coordination track record

Medium-term Projections (2-5 years)

Domain	Most Likely Outcome	Probability	Strategic Implications
Technical verification	Modest success, arms race dynamics	60%	Continued R&D investment, no single solution
Lab coordination	External oversight required	65%	Regulatory frameworks necessary
International governance	Narrow cooperation only	55%	Focus on specific risks, not comprehensive regime
Epistemic infrastructure	Chronically underfunded	70%	Accept limited scale, prioritize high-leverage applications

The resolution of these solution cruxes will fundamentally shape AI safety strategy over the next decade. If technical verification approaches prove viable, we may see an arms race between generation and detection systems. If coordination mechanisms succeed, we could see the emergence of global AI governance institutions. If they fail, we may face an uncoordinated race with significant safety risks.

Key Research Priorities

The highest-priority uncertainties requiring systematic research include:

Technical Verification Research

Systematic adversarial testing of verification systems across attack scenarios
Economic analysis comparing costs of verification vs generation at scale
Theoretical bounds on detection performance under optimal adversarial conditions
User behavior studies on provenance checking and verification adoption

Coordination Mechanism Analysis

Game-theoretic modeling of commitment mechanisms under competitive pressure
Historical analysis of coordination successes and failures in high-stakes domains
Empirical tracking of RSP implementation and compliance across labs
Regulatory effectiveness studies comparing different governance approaches

Epistemic Infrastructure Design

Hybrid system architecture for combining AI and human judgment optimally
Funding model innovation for sustainable epistemic public goods
Platform integration studies for verification system adoption
Cross-platform coordination mechanisms for epistemic infrastructure

Key Uncertainties and Strategic Dependencies

These cruxes are interconnected in complex ways that create strategic dependencies:

Technical feasibility affects coordination incentives: If verification systems work well, labs may be more willing to adopt them voluntarily
Coordination success affects infrastructure funding: Successful international cooperation could unlock government investment in epistemic public goods
Infrastructure sustainability affects technical development: Reliable funding enables long-term R&D programs for verification systems
International dynamics affect all domains: US-China competition shapes both technical development and coordination possibilities

Understanding these dependencies will be crucial for developing comprehensive solution strategies that account for the interconnected nature of technical, coordination, and infrastructure challenges.

Sources & Resources

Technical Research Organizations

Organization	Focus Area	Key Publications
DARPA↗	Semantic forensics, verification	SemaFor program↗
C2PA↗	Content provenance standards	Technical specification↗
Google DeepMind↗	Watermarking, detection	SynthID research↗

Governance and Coordination Research

Organization	Focus Area	Key Resources
GovAI↗	AI governance, coordination	Compute governance research↗
RAND Corporation↗	Strategic analysis	AI competition studies↗
CNAS↗	Security, international relations	AI security reports↗

Epistemic Infrastructure Organizations

Organization	Focus Area	Key Resources
Metaculus↗	Forecasting, prediction	AI forecasting project↗
Good Judgment↗	Superforecasting	Crowd forecasting methodology

Safety Research and Evaluation

Organization	Focus Area	Key Resources
METR↗	Third-party AI evaluations	Autonomous capability assessments
Anthropic Alignment↗	Technical alignment research	Research directions 2025↗
UK AI Safety Institute↗	Government evaluations	Evaluation approach↗

Key 2024-2025 Reports

Report	Organization	Focus
2025 AI Safety Index↗	Future of Life Institute	Industry safety practices
International AI Safety Report 2025↗	96 AI experts, 30 countries	Global safety assessment
[36fb43e4e059f0c9]	Alignment Forum	Research progress review
Mechanistic Interpretability Review↗	TMLR	Interpretability research survey
[482b71342542a659]	GovAI	Compute governance mechanisms
Global AI Governance Analysis↗	International Affairs	Governance deficit assessment