State Capacity and AI Governance

Overview

State capacity refers to a government's ability to achieve its policy goals—to actually implement and deliver, not just legislate. In the context of AI governanceCruxAI Governance and PolicyComprehensive analysis of AI governance mechanisms estimating 30-50% probability of meaningful regulation by 2027 and 5-25% x-risk reduction potential through coordinated international approaches. ...Quality: 66/100, this concept has become increasingly important as governments attempt to regulate AI systems that are orders of magnitude more complex than previous technologies.¹

The contrast between exponential growth of disruptive technology and the supply of governance mechanisms is starkest in the fields of digital governance and AI. Jennifer Pahlka, founder of Code for America and former U.S. Deputy Chief Technology Officer, argued in her 2023 book Recoding America that the separation between policy and implementation is "both false and self-defeating," rooted in an elitist view that implementation is beneath the people who make policy.² Government is trapped in rigid, compliance-driven processes rather than agile, outcome-oriented approaches. The internet era coincided with a decline in state capacity; the AI era, Pahlka argues, must see a reversal of this trend.

The challenge is acute: if governments cannot build functional IT systems—as seen in Healthcare.gov's initial collapse and state unemployment systems crashing during COVID—they face even greater challenges overseeing AI systems. Compute governancePolicyCompute GovernanceThis is a comprehensive overview of U.S. AI chip export controls policy, documenting the evolution from blanket restrictions to case-by-case licensing while highlighting significant enforcement cha...Quality: 58/100, algorithmic auditing, safety evaluations, and transparency requirements all require government staff who understand the technology. Without that capacity, regulation risks being either toothless or counterproductive.³

The AI Governance Capacity Gap

Technical Expertise Deficit

In January 2024 Senate testimony, Pahlka argued the success or failure of AI in government "comes down to how much capacity and competency we have to deploy these technologies thoughtfully."⁴ She warned against applying red tape to benign uses while lacking capacity to evaluate genuinely risky ones—a misallocation that could simultaneously slow beneficial innovation and fail to address genuine safety concerns.

The capacity gap has multiple dimensions:

Technical understanding: Regulation requires the ability to enforce—compute governancePolicyCompute GovernanceThis is a comprehensive overview of U.S. AI chip export controls policy, documenting the evolution from blanket restrictions to case-by-case licensing while highlighting significant enforcement cha...Quality: 58/100, algorithmic auditing, safety evaluations, and transparency requirements all require government staff who understand not just policy but the underlying technology. The kind of expertise needed to evaluate capability sandbaggingRiskAI Capability SandbaggingSystematically documents sandbagging (strategic underperformance during evaluations) across frontier models, finding 70-85% detection accuracy with white-box probes, 18-24% accuracy drops on autono...Quality: 67/100 or detect deceptive alignmentRiskDeceptive AlignmentComprehensive analysis of deceptive alignment risk where AI systems appear aligned during training but pursue different goals when deployed. Expert probability estimates range 5-90%, with key empir...Quality: 75/100 in AI systems is fundamentally different from traditional regulatory expertise.

Implementation capacity: Even well-crafted legislation may fail without deliberate investment in transparency and talent. The 2025 AI Safety Index from the Future of Life Institute found the industry "fundamentally unprepared for its own stated goals"—companies claim AGI within the decade but lack sufficient safety preparedness.⁵ Government oversight of this gap requires exactly the kind of technical capacity that is currently missing.

Scale mismatch: Ezra Klein has argued government needs approximately 300 excellent AI experts in advisory, regulatory, and auditing roles.⁶ This represents not just a staffing challenge but a fundamental capacity-building exercise in an area where government has historically struggled to compete with private sector compensation and prestige.

State-Level Challenges

385 AI bills were introduced across U.S. states in the 2025 legislative session.⁷ But states lack the AI expertise found in federal agencies or major private employers. Even well-crafted legislation like California SB 1047PolicySafe and Secure Innovation for Frontier Artificial Intelligence Models ActCalifornia's SB 1047 required safety testing, shutdown capabilities, and third-party audits for AI models exceeding 10^26 FLOP or \$100M training cost; it passed the legislature (Assembly 45-11, Se...Quality: 66/100 or Texas TRAIGAPolicyTexas TRAIGA Responsible AI Governance ActTRAIGA represents a state-level AI regulation focused on intent-based liability for harmful AI practices rather than comprehensive safety requirements, creating enforcement mechanisms but avoiding ...Quality: 55/100 may fail without deliberate investment in implementation capacity and talent.

The Centre for International Governance Innovation has framed the gap between AI acceleration and governance capacity as itself an existential risk vector, arguing that the "accumulative" pathway to AI catastrophe—gradual erosion of institutional capacity to manage AI-induced threats—is as dangerous as a sudden AGI scenario.⁸

The Civic Technology Ecosystem

Institutional Infrastructure

A network of organizations has emerged to build government technology capacity:

Code for America (founded 2010 by Jennifer Pahlka) helps over 100 government agencies improve digital service delivery.⁹ The organization pioneered the model of bringing technology talent into government for tours of duty.

U.S. Digital Service (USDS, founded 2014) is a White House unit focused on administration priorities, born from the Healthcare.gov rescue. It established the precedent of bringing top technologists into government for fixed terms to work on critical problems.

18F (founded 2014) was a GSA unit providing agile development and user-centered design to agencies. The unit became a training ground for government technologists, with 85% of departing 18F staff in 2023 moving to other government technology positions.¹⁰

Nava PBC builds government technology as a public benefit corporation, demonstrating that private entities can work on government technology problems while maintaining a mission-driven focus.

U.S. Digital Response (founded 2020 by Pahlka) deployed volunteer technologists during the COVID crisis, showcasing the potential for rapid capacity mobilization during emergencies.

Federation of American Scientists, where Pahlka is now a Senior Fellow, works on AI in government, connecting the civic tech ecosystem with AI governance concerns.

These organizations form a deeply interconnected ecosystem, creating a talent pipeline and shared practices for government technology work. The model they established—bringing in technologists for tours of duty, focusing on user-centered design, and building agile development capacity—represents exactly the kind of institutional capacity needed for AI oversight.

The DOGE Disruption

The Department of Government Efficiency (DOGE) initiative in 2025-2026 provides a cautionary example of what happens when technology is deployed without state capacity. DOGE fired or scattered much of the civic tech workforce, dismantled 18F, and then used AI tools (the "DOGE AI Deregulation Decision Tool") to scan 200,000 federal rules for elimination—precisely the kind of crude, capacity-poor approach that Pahlka's framework warns against.¹¹

A whistleblower complaint revealed DOGE uploaded sensitive Social Security data of 300+ million Americans to unsecured servers.¹² This illustrates the risk of deploying AI systems without the institutional capacity, expertise, and oversight structures needed to do so responsibly. The incident demonstrates how attempts to use AI to "improve" government efficiency can backfire catastrophically when they bypass the very capacity structures needed for responsible technology deployment.

The dismantling of the civic tech infrastructure is particularly concerning given the emerging AI governance challenges. The kinds of technologists USDS and 18F had been hiring—service designers, product managers, data engineers—are exactly what is needed for AI oversight, but at far greater scale.

Political Frameworks and State Capacity

State Capacity Libertarianism

Tyler Cowen coined "State Capacity Libertarianism" in 2020, arguing for a minimal but highly capable government.¹³ His position on AI risk is "radical agnosticism," but critics have argued his own state capacity framework implies that laissez-faire is inadequate for AI risk—if AGI is "effectively the most powerful weapon man has ever created," you need reliable command and control, not just market dynamics.

The framework suggests that even those skeptical of extensive government intervention should support building the capacity for effective intervention when it is necessary. The question shifts from "should government regulate AI?" to "does government have the capacity to regulate AI effectively if it needs to?"

The Governance vs. Self-Governance Debate

Holden KarnofskyPersonHolden KarnofskyHolden Karnofsky directed \$300M+ in AI safety funding through Coefficient Giving (formerly Open Philanthropy), growing the field from ~20 to 400+ FTE researchers and developing influential framewo...Quality: 40/100, co-founder of Coefficient GivingOrganizationCoefficient GivingCoefficient Giving (formerly Open Philanthropy) has directed \$4B+ in grants since 2014, including \$336M to AI safety (~60% of external funding). The organization spent ~\$50M on AI safety in 2024...Quality: 55/100, has argued "there's no way to get to an actual low-level risk from AI without government policy playing an important role" and that escaping race dynamics requires universal rules.¹⁴ His preferred sequence: voluntary company-level commitments first (like AnthropicOrganizationAnthropicComprehensive reference page on Anthropic covering financials (\$380B valuation, \$19B ARR), safety research (Constitutional AI, mechanistic interpretability, model welfare), governance (LTBT struc...Quality: 74/100's Responsible Scaling Policy), then government regulation builds on top of that foundation.

This approach acknowledges both the need for government capacity and the current limitations. Industry self-governance can establish practices and norms that later regulation can formalize and universalize—but only if government has the capacity to understand, evaluate, and enforce those norms when they become regulation.

The Cascade of Rigidity

The Niskanen Center has published Pahlka's work on "AI meets the cascade of rigidity," arguing that the same institutional dynamics that trapped government in waterfall-mode IT will hamper AI adoption and oversight unless actively reformed.¹⁵ The problem is not just lack of technical expertise but the structural impediments that make it difficult for government to act with the speed and flexibility that AI governance may require.

Government procurement rules, personnel systems, and accountability structures were designed for an earlier era. They create what Pahlka calls a "cascade of rigidity"—each layer of process and compliance adding more constraints, until the system becomes nearly incapable of adapting to new challenges. This suggests that building AI governance capacity requires not just hiring more technical experts but fundamentally reforming how government operates.

Building Capacity for AI Oversight

Institutional Reform

Pahlka has called for a "clear-eyed leapfrog" into a government operating model fit for the AI era, rather than bolting AI onto industrial-era bureaucracy.¹⁶ This would require:

Process reform: Moving from rigid, compliance-driven processes to agile, outcome-oriented approaches. This doesn't mean abandoning accountability but reimagining how accountability works in a fast-moving technical domain.

Talent pipeline: Creating career paths that make government technology work attractive to top technical talent. This includes competitive compensation, opportunities for meaningful impact, and pathways between government and private sector that don't stigmatize either direction of movement.

Technical infrastructure: Building the institutional capacity to understand and evaluate AI systems. This includes compute resources for independent testing, access to training data and model internals, and the methodological expertise to conduct meaningful audits.

Regulatory experimentation: Developing the capacity to test different governance approaches and learn from results, rather than locking in approaches that may prove inadequate.

Scale and Urgency

Klein and Derek Thompson's "Abundance" framework argues for building government effectiveness as a prerequisite for managing technological change.¹⁷ This suggests the capacity-building challenge is not peripheral but central to navigating the AI transition safely.

The timeline is challenging. If AI capabilities continue to advance rapidly, the window for building adequate governance capacity may be narrow. Yet capacity-building is inherently slow—it requires not just hiring people but building institutions, establishing practices, and creating the organizational culture that enables effective action.

Key Uncertainties

Capacity threshold: How much state capacity is actually needed for safe AI governance? Klein's estimate of 300 excellent experts may be orders of magnitude too low if AI systems become as complex and consequential as some predict. Alternatively, with the right institutional design, effective governance might be possible with fewer but better-positioned people.

Speed of capacity building: Can government capacity grow fast enough to keep pace with AI capabilities? The civic tech ecosystem took over a decade to establish, and DOGE disrupted much of it in months. Whether capacity-building or capacity-destruction is faster has major implications for AI safety trajectories.

Institutional design: What organizational structures enable effective AI governance? Should capacity be centralized in specialized agencies, distributed across existing regulators, or some hybrid? Different structures have different strengths and weaknesses for handling fast-moving technical challenges.

International dimension: While much discussion focuses on U.S. state capacity, AI governance is inherently international. How do capacity disparities between countries affect global governance prospects? Can international institutions provide capacity where national governments lack it?

Public vs. private capacity: Should government build in-house capacity or rely on partnerships with private sector organizations (like Nava PBC) or academic institutions? Each approach has different scaling properties and different risks.

Measurement and accountability: How do we measure whether governance capacity is adequate? What metrics distinguish between capacity that looks impressive on paper and capacity that can actually implement and enforce policy effectively?

Implications for AI Safety Strategy

The state capacity framing suggests several strategic considerations for AI safety work:

Capacity-building as intervention: Efforts to build government technical capacity may be as important as research on AI alignment or safety techniques. Without implementation capacity, even excellent policy ideas remain merely theoretical.

Sequencing matters: Karnofsky's preferred sequence—voluntary company commitments followed by government regulation—implicitly acknowledges that government capacity needs time to develop. This suggests AI safety strategies should consider not just what governance is ultimately needed but what trajectory of capacity-building makes that governance achievable.

Field-building scope: AI safety field-buildingCruxAI Safety Field Building and CommunityGrowing the AI safety research community through funding, training, and outreach0 typically focuses on research capacity, but governance capacity may be equally important. The skills needed—policy analysis, institutional design, regulatory expertise, technical-policy translation—are different from traditional AI safety research but potentially as important for outcomes.

Vulnerability to disruption: The DOGE example shows that governance capacity can be destroyed faster than it can be built. This creates a ratchet effect—periods of capacity-destruction may be difficult to reverse. This suggests that protecting existing governance institutions may be as important as building new ones.

Crisis as opportunity: U.S. Digital Response's founding during COVID shows that crises can enable rapid capacity mobilization. If an AI-related incident creates demand for governance capacity, having the organizational templates and talent networks ready to activate quickly could be crucial.

Criticisms and Limitations

Over-emphasis on Technical Capacity

Some critics argue the focus on technical capacity misses that AI governance challenges are fundamentally political, not technical. Having experts who understand transformer architectures doesn't necessarily help resolve value conflicts about how AI should be governed or whose interests should be prioritized.

Elite Technocracy Concerns

The civic tech model of bringing in outside experts for tours of duty has been criticized as elitist and potentially undemocratic. Career civil servants may have institutional knowledge that tour-of-duty technologists lack. There's a tension between the need for technical expertise and the democratic legitimacy of governance institutions.

Narrow Window Assumption

The framing assumes there's a window during which capacity can be built before AI capabilities reach concerning levels. If AI capabilities advance faster than expected, or if the threshold for dangerous capabilities is lower than assumed, capacity-building approaches may be too slow to matter.

International Blind Spots

Most discussion of state capacity and AI governance focuses on the United States. But AI development is international, and U.S. state capacity may be irrelevant if other countries lack similar capacity and become sites of AI development or deployment.

Sources