Large Language Models

Quick Assessment

Key Links

Overview

Large Language Models (LLMs) are transformer-based neural networks trained on vast text corpora using next-token prediction. Despite their deceptively simple training objective, LLMs exhibit sophisticated emergent capabilitiesRiskEmergent CapabilitiesEmergent capabilities—abilities appearing suddenly at scale without explicit training—pose high unpredictability risks. Wei et al. documented 137 emergent abilities; recent models show step-functio...Quality: 61/100 including reasoning, coding, scientific analysis, and complex task execution. These models have transformed abstract AI safety discussions into concrete, immediate concerns while providing the clearest path toward artificial general intelligence.

The core insight underlying LLMs is that training a model to predict the next word in a sequence—a task achievable without labeled data—produces internal representations useful for a wide range of downstream tasks. This approach was explored in early unsupervised pretraining work from OpenAI in 2018.¹ OpenAI's GPT-2 (2019) then demonstrated coherent multi-paragraph generation at scale, showing that larger models trained on more data produced qualitatively stronger outputs.² An earlier indicator that such models form interpretable internal structure was the discovery of an "unsupervised sentiment neuron" in 2017, which emerged without any sentiment-specific supervision.³

Current frontier models—including GPT-4o, Claude Opus 4.5, Gemini 2.5 Pro, and Llama 4—demonstrate near-human or superhuman performance across diverse cognitive domains. Training runs for leading frontier systems reportedly consume hundreds of millions of dollars,⁴ and model parameter counts have reached into the hundreds of billions to trillions.⁵ These substantial computational investments have shifted AI safety from theoretical to practical urgency. The late 2024–2025 period marked a paradigm shift toward inference-time compute scaling, with reasoning models such as OpenAI's o1 and o3 achieving higher performance on reasoning benchmarks by allocating more compute at inference rather than only at training time.⁶

A parallel development is the rapid growth of the open-weight ecosystem. Meta's Llama family has grown substantially since its initial release, with Meta reporting over a billion downloads and more than ten times the developer activity compared to 2023.⁷ Google's Gemma models—including Gemma 3 and the mobile-first Gemma 3n variants—have provided the safety research community with accessible architectures for mechanistic interpretability work.⁸ This open/closed model convergence has implications for both capability diffusion and the tractability of safety interventions.

Capability Architecture

The diagram below maps the flow from training through inference to observed capabilities. Capabilities are grouped by which inference regime produces them; the distinction between "standard" and "search-augmented" inference is a key axis along which safety-relevant behaviors (extended planning, autonomous task execution) emerge.

Note: Capabilities in the "Emergent Capabilities" subgraph are descriptive, not evaluative. Safety implications of each capability class are discussed in the Concerning Capabilities Assessment and Safety-Relevant Positive Capabilities sections below.

Risk Assessment

Capability Progression Timeline

Primary sources: OpenAI model announcements, Anthropic model cards, Google DeepMind blog.

Benchmark Performance Comparison (2024–2025)

Sources: OpenAI o1 system card⁶, ARC Prize o3 analysis⁷, Anthropic Claude 3.5 model card⁵.

The o3 results represent a qualitative shift: o3 achieved nearly human-level performance on ARC-AGI (87.5%) versus a ~85% human baseline⁷, a benchmark specifically designed to test general reasoning rather than pattern matching. On FrontierMath, o3 reportedly solved 25.2% of problems compared to o1's 2%—a roughly 12x improvement that suggests reasoning capabilities may be scaling faster than expected⁷. However, on the harder ARC-AGI-2 benchmark, o3 scores only ~3% compared to ~60% for average humans⁷, revealing significant limitations in truly novel reasoning tasks.

Scaling Laws and Predictable Progress

Core Scaling Relationships

Research by Kaplan et al. (2020)↗📄 paper★★★☆☆arXivKaplan et al. (2020)Jared Kaplan, Sam McCandlish, Tom Henighan et al. (2020)capabilitiestrainingcomputellm+1Source ↗¹ and refined by Hoffmann et al. (2022)↗📄 paper★★★☆☆arXivHoffmann et al. (2022)Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch et al. (2022)capabilitiestrainingevaluationcompute+1Source ↗² demonstrates robust mathematical relationships governing LLM performance:

Sources: Chinchilla paper²; Scaling Laws for Neural Language Models¹

According to Epoch AIOrganizationEpoch AIEpoch AI maintains comprehensive databases tracking 3,200+ ML models showing 4.4x annual compute growth and projects data exhaustion 2026-2032. Their empirical work directly informed EU AI Act's 10...Quality: 51/100 research,³ approximately two-thirds of LLM performance improvements over the last decade are attributable to increases in model scale, with training techniques contributing roughly 0.4 orders of magnitude per year in compute efficiency.³ The cost of training frontier models has grown by reportedly 2.4x per year since 2016, with the largest models projected to exceed $1B by 2027, according to Epoch AI cost analyses.⁴

A related phenomenon is the emergence of new scaling regimes beyond training compute. Research published in 2025 finds that emergent capability thresholds are sensitive to random factors including data ordering and initialization seeds,⁵ suggesting that the apparent sharpness of emergence in aggregate curves may partly reflect averaging over many random runs with different thresholds rather than a clean phase transition.⁵

The Shift to Inference-Time Scaling (2024–2025)

The o1 and o3 model families introduced a new paradigm: inference-time compute scaling. Rather than only scaling training compute, these models allocate additional computation at inference time through extended reasoning chains and search procedures.⁶

This shift has implications for AI safety: inference-time scaling allows models to "think longer" on hard problems, potentially achieving strong performance on specific tasks while maintaining manageable training costs. According to some sources, o1 is approximately 6x more expensive and 30x slower than GPT-4o per query.⁷ The Stanford AI Index 2025 cites the RE-Bench evaluation finding that in short time-horizon settings (2-hour budget), top AI systems score 4x higher than human experts, but as the time budget increases to 32 hours, human performance surpasses AI by roughly 2 to 1.⁸

Emergent Capability Thresholds

Research documented in a 2025 emergent abilities survey⁷ finds that emergent abilities depend on multiple interacting factors: scaling up parameters or depth lowers the threshold for emergence but is neither necessary nor sufficient alone—data quality, diversity, training objectives, and architecture modifications also matter significantly.⁷ Emergence aligns more closely with pre-training loss landmarks than with sheer parameter count; smaller models can match larger ones if training loss is sufficiently reduced.⁷

According to the Stanford AI Index 2025,⁸ benchmark performance improved substantially in a single year: scores rose by 18.8, 48.9, and 67.3 percentage points on MMMU, GPQA, and SWE-bench respectively.⁸ The gap between leading US and Chinese models also narrowed—from 17.5 to 0.3 percentage points on MMLU—over the same period.⁸

Safety implication: As AI systems gain autonomous reasoning capabilities, they also develop behaviors relevant to safety evaluation, including goal persistence, strategic planning, and the capacity for deceptive alignmentRiskDeceptive AlignmentComprehensive analysis of deceptive alignment risk where AI systems appear aligned during training but pursue different goals when deployed. Expert probability estimates range 5-90%, with key empir...Quality: 75/100. OpenAI's o3-mini reportedly became the first AI model to receive a "Medium risk" classification for Model Autonomy under internal capability evaluation frameworks.⁹

RLHF and Alignment Training Foundations

A key technique underlying modern aligned LLMs is Reinforcement Learning from Human Feedback (RLHF), which trains a reward model on human preference comparisons and uses it to fine-tune Large Language Models outputs via RL. The foundational application of this approach to Large Language Models was demonstrated in Fine-Tuning GPT-2 from Human Preferences (Ziegler et al., 2019), which showed that human feedback could steer model behavior on stylistic tasks. This was later scaled to instruction-following via InstructGPT and then to Claude's Constitutional AI framework.

A significant limitation is that RLHF-trained models can exhibit sycophancyRiskSycophancySycophancy—AI systems agreeing with users over providing accurate information—affects 34-78% of interactions and represents an observable precursor to deceptive alignment. The page frames this as a...Quality: 65/100—systematically agreeing with user beliefs rather than providing accurate responses—because human raters often prefer confident, agreeable answers. Recent work on multi-objective alignment for psychotherapy contexts and policy-constrained alignment frameworks like PolicyPad (2025) explore how to balance multiple competing alignment objectives.

Multilingual alignment gap: A consistent finding across alignment research is that safety training applied primarily in English does not reliably transfer to other languages. Align Once, Benefit Multilingually (2025) proposes methods to enforce multilingual consistency in safety alignment by training on cross-lingual consistency losses. Related work on Helpful to a Fault (2025) measures illicit assistance rates across 40+ languages in multi-turn interactions, finding that multilingual agents provide more potentially harmful assistance than English-only evaluations would suggest, particularly in low-resource languages where safety fine-tuning data is sparse.

Major 2025 Model Releases

GPT-5 and the GPT-5 Family

GPT-5, released in May 2025, represents OpenAIOrganizationOpenAIComprehensive organizational profile of OpenAI documenting evolution from 2015 non-profit to Public Benefit Corporation, with detailed analysis of governance crisis, 2024-2025 ownership restructuri...Quality: 62/100's integration of reasoning and general capability into a single unified model, replacing the separate GPT-4o and o1 product lines. According to the GPT-5 System Card, GPT-5 achieves the highest scores to date on GPQA Diamond, SWE-bench Verified, and MMLU among OpenAI models at the time of release.

Key developments in the GPT-5 family:

• GPT-5.1: Released for developers in mid-2025, optimized for conversational applications and described as "smarter, more conversational" with improved instruction-following. Cursor and Tolan reported substantial gains in their agentic pipelines.
• gpt-oss-120b and gpt-oss-20b: Open-weight models released by OpenAI, with model cards published for both sizes. These represent a shift toward open-weight strategy alongside closed frontier models.
• GPT Realtime API: Extended with real-time audio dialog capabilities, building on the GPT-4o voice capabilities introduced in Hello GPT-4o (2024).

The GPT-5 System Card documents safety evaluations including assessments of CBRN uplift risk, cyberoffense capabilities, and persuasion. An Addendum on Sensitive Conversations addresses handling of mental health, self-harm, and politically contentious topics, noting both improvements in refusal precision and remaining cases where the model provides responses that require strengthening.

Gemini 2.5 Family

Google DeepMind's Gemini 2.5 family, released across March–June 2025, introduced several models:

Gemini 2.5: Our Most Intelligent AI Model describes the Pro variant as achieving leading performance on coding benchmarks including LiveCodeBench and outperforming GPT-4o on MMLU at release. Gemini 2.5 Flash-Lite was made production-ready in June 2025 with a focus on throughput-sensitive applications.

The 2.5 family also introduced native thinking models—models that produce explicit chain-of-thought reasoning before answering—across both Pro and Flash tiers. Advanced audio dialog and generation with Gemini 2.5 extended audio-native generation capabilities.

Gemma 3 and Open-Weight Models

Google's Gemma 3 family, released in 2025, provides open-weight models ranging from 1B to 27B parameters optimized for single-accelerator deployment. The Gemma 3 270M variant targets edge and mobile deployment. Gemma 3n introduced a mobile-first architecture with selective parameter activation for on-device inference.

MedGemma, released in mid-2025, provides open health-specific models demonstrating LLM application in clinical reasoning. A Gemma-based model contributed to discovery of a potential cancer therapy pathway, illustrating scientific research acceleration potential.

T5Gemma introduced encoder-decoder variants of the Gemma architecture, enabling use cases where separate encoding and decoding is beneficial (e.g., retrieval-augmented generation, classification tasks).

Llama 4 and the Open-Weight Ecosystem

Meta's Llama 4 Herd, announced at LlamaCon in April 2025, represents a shift to natively multimodal architecture using a mixture-of-experts design. Llama 4 Scout and Llama 4 Maverick support image, video, and text inputs from the base model level. Meta reported over 10x growth in Llama usage since 2023, with the model family becoming a reference implementation for open-weight AI development.

Key safety implications of the open-weight ecosystem:

• Fine-tuning safety guardrails out of open-weight models remains tractable for technically sophisticated users
• Mechanistic interpretability research benefits from open weights (e.g., Gemma Scope 2, described below)
• Governance frameworks targeting API access do not apply to locally deployed open-weight models

Mechanistic Interpretability Advances

Mechanistic interpretabilityApproachMechanistic InterpretabilityMechanistic interpretability aims to reverse-engineer neural networks to understand internal computations, with \$100M+ annual investment across major labs. Anthropic extracted 30M+ features from C...Quality: 59/100 research—which seeks to understand the internal computations of neural networks in human-interpretable terms—has accelerated substantially with the availability of open-weight models and new tooling.

Gemma Scope 2

Gemma Scope 2 is a suite of sparse autoencoders (SAEs) trained on Gemma 3 models, released by Google DeepMind to support interpretability research on complex language model behaviors. Building on the original Gemma Scope release for Gemma 2, Gemma Scope 2 provides SAEs at multiple layers and widths, enabling decomposition of model activations into human-interpretable features.

Gemma Scope 2 supports research into:

• Feature geometry and polysemanticity in larger models
• Cross-layer feature interactions and information flow
• Identification of features relevant to safety-relevant behaviors (deception, refusal, sycophancy)

Language Models Explaining Neurons

Language Models Can Explain Neurons in Language Models (Bills et al., 2023) demonstrated that GPT-4 can generate natural language explanations of GPT-2 neurons with higher validity than human-written explanations. This opened a scalable pathway for automated interpretability: using more capable models to explain less capable ones. Subsequent work has extended this to sparse autoencoder features and cross-model explanation transfer.

Activation Steering and Causal Intervention

Activation steering—injecting vectors into model residual streams to steer behavior—has become a primary tool for behavioral intervention research. Recent work has refined understanding of when and why steering succeeds:

• Surgical Activation Steering via Generative Causal Mediation (2025) demonstrates that steering effectiveness depends on correctly identifying the causal pathway through which a concept influences output, rather than simply finding directions in activation space. Steering at incorrect layers or attention heads produces unreliable or null effects.
• Mechanistic Indicators of Steering Effectiveness in Large Language Models (2025) identifies model-internal signals that predict whether a given steering vector will successfully alter behavior, enabling more principled selection of intervention targets.
• Do Personality Traits Interfere? Geometric Limitations of Steering in Large Language Models (2025) finds that personality-related features form geometric structures in activation space that can interfere with unrelated steering directions, suggesting that alignment-relevant features are not always cleanly separable.
• Does GPT-2 Represent Controversy? A Small Mech Interp Investigation (2024) provides a case study of applying mechanistic interpretability methods to identify controversy-related representations in GPT-2, illustrating the workflow for small-scale interpretability research.
• Features as Rewards: Scalable Supervision for Open-Ended Tasks via Interpretability (2025) proposes using interpretability-discovered features as reward signals for RLHF, bypassing the need for human labeling of open-ended tasks.
• Causality is Key for Interpretability Claims to Generalise (2025) argues that interpretability claims must be evaluated using causal interventions rather than correlation alone, since correlational analyses can identify spurious structure that does not causally influence model behavior.

Research Platform Value of Open-Weight Models

Hallucination and Factuality

LLM hallucination—the generation of plausible-sounding but factually incorrect or unsupported content—remains a central reliability and safety challenge. Hallucination rates vary substantially by task, model, and measurement methodology, with published estimates ranging from roughly 8% to 45% depending on benchmark design and model version.¹⁰

Factuality Benchmarking

The FACTS Grounding benchmark¹¹ introduced by Google DeepMind specifically addresses the challenge of evaluating long-form generation against reference documents, distinguishing between claims grounded in provided source material and claims introduced without basis. This is particularly relevant for retrieval-augmented generation (RAG) systems, where the model has access to retrieved context but may still generate unsupported claims.

CheckIfExist (2025)¹² addresses citation hallucination specifically—the generation of plausible-looking but non-existent citations, which poses particular risks in legal, medical, and academic contexts. The benchmark finds that citation hallucination rates remain substantial even for frontier models, and that RAG systems can still hallucinate citations from retrieved documents by misattributing or confabulating specific reference details.

Why Models Hallucinate

OpenAI's explainer on why language models hallucinate¹³ identifies several contributing mechanisms:

1. Training objective mismatch: Next-token prediction rewards coherent text, not factual accuracy
2. Knowledge compression: Models must compress world knowledge into fixed-weight representations, leading to lossy encoding
3. Context-weight tension: Models may blend retrieved context with parametric knowledge, producing hybrid outputs that are faithful to neither
4. Sycophancy pressure: RLHF can train models to confirm user beliefs rather than correct factual errors, since raters may prefer agreeable responses
5. Calibration failure: Models often express high confidence in incorrect claims, reducing the signal value of expressed uncertainty

Hallucination rates are not monotonically improved by scale: models engineered for massive context windows do not consistently achieve lower hallucination rates than smaller counterparts, and increased model size can increase the confidence of hallucinated outputs without reducing their frequency. This suggests hallucination is partly an architectural and training-objective issue rather than purely a capacity limitation.

Deception and Truthfulness

According to some sources, even recent frontier models have hallucination rates exceeding 15% when asked to analyze provided statements, and approximately 1 in 6 AI responses in legal contexts reportedly contain citation hallucinations.¹⁴ The wide variance across benchmarks reflects both genuine model differences and definitional variation in what constitutes a hallucination.

Safety-Relevant Positive Capabilities

Interpretability Research Platform

Constitutional AI and Value Learning

Anthropic's Constitutional AI↗📄 paper★★★★☆AnthropicAnthropic's Work on AI SafetyAnthropic conducts research across multiple domains including AI alignment, interpretability, and societal impacts to develop safer and more responsible AI technologies. Their w...alignmentinterpretabilitysafetysoftware-engineering+1Source ↗ demonstrates approaches to value alignment through self-critique, principle-following, and preference learning. The specific success rates for these techniques vary considerably across evaluations, tasks, and model versions; the figures below represent approximate reported ranges rather than settled benchmarks, and should be treated with caution.

Scalable OversightSafety AgendaScalable OversightProcess supervision achieves 78.2% accuracy on MATH benchmarks (vs 72.4% outcome-based) and is deployed in OpenAI's o1 models, while debate shows 60-80% accuracy on factual questions with +4% impro...Quality: 68/100 Applications

Modern LLMs enable several approaches to AI safety through automated oversight:

• Output evaluation: AI systems critiquing other AI outputs. Agreement rates with human evaluators vary substantially by task and domain; no single well-characterized aggregate figure covers the literature.
• Red-teaming: Automated discovery of failure modes and adversarial inputs.
• Safety monitoring: Real-time analysis of AI system behavior patterns.
• Research acceleration: AI-assisted safety research and experimental design.
• Content moderation: OpenAI has published work on using GPT-4 for content moderation, describing how LLM-based moderation can operate at scale and reduce human labeler exposure to harmful content.

Concerning Capabilities Assessment

Persuasion and Manipulation

Modern LLMs demonstrate persuasion capabilities that raise questions for democratic discourse and individual autonomy:

According to some sources, frontier LLMs can substantially increase human agreement rates through targeted persuasion techniques, raising concerns about consensus manufacturingRiskAI-Powered Consensus ManufacturingConsensus manufacturing through AI-generated content is already occurring at massive scale (18M of 22M FCC comments were fake in 2017; 30-40% of online reviews are fabricated). Detection systems ac...Quality: 64/100; however, the specific figure of 82% attributed to Anthropic research has not been independently verified against a primary source and should be treated with caution.¹⁵ The ARC EvaluationsOrganizationARC EvaluationsOrganization focused on evaluating AI systems for dangerous capabilities. Now largely absorbed into METR. (2025) specifically evaluates persuasion and resistance capabilities in LLMs through adversarial resource extraction games, finding that frontier models can maintain persuasive pressure across extended multi-turn interactions.¹⁶

Multilingual Safety Gaps

A consistent finding across the research literature is that safety alignment applied primarily in English does not reliably transfer to other languages. Helpful to a Fault (2025) measured illicit assistance rates for multi-turn, multilingual LLM agents across 40+ languages, finding that:¹⁷

• Models provide meaningfully more potentially harmful assistance in non-English languages
• The gap is largest for low-resource languages with sparse alignment training data
• Multi-turn interactions elicit more harmful assistance than single-turn, with each turn potentially eroding earlier refusals

This suggests that multilingual deployment of LLMs creates safety gaps that single-language evaluation would miss, a concern reportedly noted in the GPT-5 System Card.⁹

Cybersecurity Capabilities and Refusal Frameworks

LLMs' code generation and vulnerability analysis capabilities create dual-use risks in cybersecurity:

• Models can reportedly assist with vulnerability identification, exploit development, and attack planning at varying levels depending on the specificity of the request
• A Content-Based Framework for Cybersecurity Refusal Decisions (2025) proposes taxonomizing cybersecurity requests by content type (reconnaissance vs. exploitation vs. malware development) rather than binary harmful/safe classification, enabling more precise refusal decisions that maintain legitimate educational and defensive use¹⁸
• SecCodeBench-V2 (2025) provides updated benchmarks for evaluating security-relevant code generation, reportedly finding continued improvement in both benign and potentially harmful code generation capabilities¹⁹

Autonomous Capabilities

Current LLMs demonstrate increasing levels of autonomous task execution:

• Web browsing: GPT-4 class models can reportedly navigate websites, extract information, and interact with web services
• Code execution: Models can write, debug, and iteratively improve software across extended sessions
• API integration: Tool use across multiple digital platforms; OpenAI's Shipping Smarter Agents with Every New Model describes the company's agent capability roadmap²⁰
• Goal persistence: Basic ability to maintain objectives across extended multi-turn interactions, relevant to schemingRiskSchemingScheming—strategic AI deception during training—has transitioned from theoretical concern to observed behavior across all major frontier models (o1: 37% alignment faking, Claude: 14% harmful compli...Quality: 74/100 risk evaluation
• Memory across sessions: MemoryArena (2025) benchmarks agent memory in interdependent multi-session tasks, finding that frontier models maintain task state imperfectly across sessions but with increasing reliability in newer model versions²¹

Fundamental Limitations

What Doesn't Scale Automatically

Current Performance Gaps

Despite strong benchmark performance, significant limitations remain:

• Hallucination rates: According to some sources, hallucination rates range from roughly 8–45% depending on task and model, with high variance across domains¹
• Inconsistency: High variance in responses to equivalent prompts; Evidence for Daily and Weekly Periodic Variability in GPT-4o Performance (2025)² documents time-dependent performance variation that is difficult to explain and control for in evaluations
• Context limitations: Models struggle with very long-horizon reasoning despite large context windows (reportedly exceeding 1 million tokens in some deployments); long-context retrieval accuracy degrades for content positioned in the middle of the context window
• Novel problem solving: According to reporting at the time of its release, o3 achieved approximately 87.5% on ARC-AGI; however, this reportedly required high-compute settings, and real-world novel reasoning remains challenging³. On the harder ARC-AGI-2 benchmark, o3 reportedly scored around 3% while average humans scored approximately 60%⁴
• Benchmark vs. real-world gap: Research using the QUAKE benchmark reportedly found that frontier LLMs average just 28% pass rate on practical tasks, despite high scores on standard benchmarks⁵
• Long-tail knowledge: Long-Tail Knowledge in Large Language Models (2025)⁶ documents that rare facts and edge cases are disproportionately hallucinated, with performance degrading sharply below certain frequency thresholds in training data
• False belief reasoning: Language Statistics and False Belief Reasoning: Evidence from 41 Open-Weight LMs (2025)⁷ finds that models rely heavily on surface language statistics rather than genuine theory-of-mind reasoning when solving false belief tasks, raising questions about claimed social cognition capabilities

Note that models engineered for massive context windows do not consistently achieve lower hallucination rates than smaller counterparts, suggesting performance depends more on architecture and training quality than capacity alone.

Current State and 2025-2030 Trajectory

Key 2024-2025 Developments

One of the most significant trends is the emergence of agentic AI—LLM-powered systems that can make decisions, interact with tools, and take actions without constant human input. This represents a qualitative shift from chat interfaces to autonomous systems capable of extended task execution. OpenAI's Spring Update and subsequent Shipping Smarter Agents posts describe investment in memory, tool use, and multi-model orchestration as the primary near-term product directions.

Near-term Outlook (2025-2026)

Medium-term Outlook (2026-2030)

Expected developments include potential architectural innovations beyond standard transformer attention, deeper integration with robotics platforms, and continued capability improvements. Key uncertainties include whether current scaling approaches will continue yielding improvements and the timeline for artificial general intelligence.

Data constraints: According to Epoch AI projections, high-quality training data could become a significant bottleneck this decade, particularly if models continue to be overtrained. For AI progress to continue into the 2030s, either new sources of data (synthetic data, multimodal data) or less data-hungry techniques must be developed. Can Generative AI Survive Data Contamination? (2025) analyzes the theoretical guarantees available under contaminated recursive training—where models are trained on AI-generated data—finding that certain forms of contamination lead to systematic performance degradation rather than catastrophic collapse, but that the long-run effects remain analytically uncertain.

Key Uncertainties and Research Cruxes

Fundamental Understanding Questions

• Intelligence vs. mimicry: Extent of genuine understanding vs. sophisticated pattern matching, with False Belief Reasoning research (2025) suggesting current models rely heavily on statistical correlates of reasoning rather than causal models
• Emergence predictability: Whether capability emergence can be reliably forecasted; Random Scaling of Emergent Capabilities (2025) suggests randomness in training may cause more gradual aggregate emergence than apparent
• Architectural limits: Whether transformers can scale to AGI or require fundamental innovations
• Alignment scalability: Whether current safety techniques work for superhuman systems

Safety Research Priorities

Timeline Uncertainties

Current expert surveys show wide disagreement on AGI timelines, with median estimates ranging from 2027 to 2045. This uncertainty stems from:

• Unpredictable capability emergence patterns
• Unknown scaling law continuation
• Potential architectural innovations
• Economic and resource constraints
• Data availability bottlenecks

The o3 results on ARC-AGI (87.5%, approaching human baseline of ~85%) have intensified debate about whether we are approaching AGI sooner than expected. Critics note that high-compute inference settings make this performance expensive and slow, and that benchmark performance may not translate to general real-world capability. The ARC-AGI-2 results (o3 at 3%, average humans at 60%) illustrate that performance on designed-to-be-robust benchmarks still reveals substantial gaps.

Sources & Resources

Academic Research

Model and System Cards

Organizations and Research Groups

Policy and Governance Resources