Good Judgment

📋Page Status

Page Type:ContentStyle Guide →Standard knowledge base article

Quality:50 (Adequate)⚠️

Importance:45 (Reference)

Last edited:2026-02-01 (5 days ago)

Words:4.4k

Structure:

📊 2📈 0🔗 9📚 84•0%Score: 12/15

LLM Summary:Good Judgment Inc. is a commercial forecasting organization that emerged from successful IARPA research, demonstrating that trained 'superforecasters' can outperform intelligence analysts and prediction markets by 30-72%. While not directly focused on AI safety, their methodology for identifying forecasting talent and improving prediction accuracy has potential applications for AI risk assessment.

Issues (2):

QualityRated 50 but structure suggests 80 (underrated by 30 points)
Links4 links could use <R> components

Quick Assessment

Aspect	Assessment
Type	Commercial forecasting organization
Founded	2015 (from Good Judgment Project, 2011-2015)
Key Innovation	Identification and employment of “superforecasters”
Track Record	30-72% more accurate than competitors; outperformed intelligence analysts by 30%
Primary Focus	Geopolitical forecasting, strategic risk assessment
Key Products	FutureFirst™, training workshops, custom forecasts
Funding	Government contracts, commercial clients (energy, finance, nonprofits)
AI Safety Relevance	Limited direct involvement; potential application to AI risk forecasting

Key Links

Source	Link
Official Website	gjopen.com
Wikipedia	en.wikipedia.org

Overview

Good Judgment Inc. is a commercial forecasting organization that emerged from the Good Judgment Project (GJP), a research initiative that ran from 2011 to 2015 as part of the Intelligence Advanced Research Projects Activity (IARPA) Aggregative Contingent Estimation (ACE) tournament.¹ The organization’s core innovation was the identification of “superforecasters”—individuals who demonstrate exceptional accuracy in probabilistic forecasting on geopolitical and global events, often outperforming professional intelligence analysts with access to classified information.²

Founded by Philip Tetlock and Barbara Mellers of the University of Pennsylvania, Good Judgment pioneered methods for crowd-sourced forecasting that combine amateur forecasters, bias training, and aggregation algorithms to produce remarkably accurate predictions.³ During the IARPA tournament, the Good Judgment Project won outright with performance 35-72% better than rival teams and more than 30% better than intelligence community analysts.⁴ The project discovered that certain individuals—dubbed superforecasters—could consistently make more accurate predictions than both domain experts and prediction markets.

After the research phase concluded in 2015, Good Judgment transitioned to a commercial model, offering forecasting services to organizations across government, energy, finance, and nonprofit sectors.⁵ The company maintains a global network of superforecasters across six continents who provide 24/7 crowd-sourced insights on strategic questions through platforms like FutureFirst™.⁶ Good Judgment also offers training programs to help organizations develop internal forecasting capabilities, emphasizing probabilistic thinking, bias correction, and accountable decision-making.

History

Origins in Academic Research

The roots of Good Judgment trace back to Philip Tetlock’s 1984 research on forecasting tournaments involving over 250 experts in political and economic trends.⁷ This early work revealed a surprising finding: domain expertise did not strongly correlate with predictive accuracy. Tetlock’s book Expert Political Judgment documented systematic failures in expert predictions and laid the groundwork for investigating alternative approaches to forecasting.

In 2011, following intelligence community failures to anticipate major geopolitical events, IARPA launched the Aggregative Contingent Estimation (ACE) tournament to identify better forecasting methods.⁸ Tetlock and Barbara Mellers formed the Good Judgment Project as a competing team, recruiting amateur forecasters and testing various techniques including bias tutorials, aggregation algorithms, and team deliberation structures.

The IARPA Tournament (2011-2015)

The Good Judgment Project’s performance in the ACE tournament was exceptional. During its first year in 2011, the project recruited forecasters like Jean-Pierre Beugoms and generated over 1 million forecasts across 500 questions ranging from Venezuelan gas subsidies to North Korean politics.⁹ The questions were carefully designed as verifiable predictions with clear resolution criteria, scored using Brier scores to measure probabilistic accuracy.

By 2012, GJP had identified a subset of elite forecasters and organized them into teams of approximately 12 individuals.¹⁰ The median forecasts from these superforecaster teams proved 35-72% more accurate than competing teams, demonstrating that simple aggregation of skilled forecasters could rival sophisticated algorithms. The project’s success was so pronounced that by summer 2013, GJP became the sole IARPA-funded team and gained access to the Integrated Conflict Early Warning System.¹¹

The tournament continued through 2015, with IARPA ending it early due to GJP’s dominance.¹² Throughout the five-year period, the project produced over 1 million individual forecasts, identified consistent patterns in forecasting skill, and demonstrated that superforecasters could maintain their accuracy advantage over time. Notably, superforecasters proved more accurate than professional intelligence analysts with access to classified information—a finding that challenged conventional assumptions about the value of domain expertise and secret intelligence.¹³

Commercialization and Growth

Following the conclusion of the IARPA research in fall 2015, Good Judgment Inc. launched as a commercial entity.¹⁴ The company began hiring professional superforecasters like Jean-Pierre Beugoms, who transitioned from volunteer forecaster to full-time professional.¹⁵ Good Judgment also established Good Judgment Open (GJ Open), a public forecasting platform that serves both as a recruitment pipeline for identifying new superforecasters and as a training ground for developing forecasting skills.¹⁶

The commercial organization expanded beyond geopolitics to address questions in finance, energy, public health, and organizational strategy.¹⁷ Good Judgment developed proprietary tools like FutureFirst™, which provides clients with continuous forecasting insights by assigning approximately 40 superforecasters to each client question.¹⁸ The company also built a training business, offering workshops and seminars on probabilistic thinking, bias correction, and forecasting methodology to government agencies, corporations, and nonprofits across the United States, United Kingdom, Netherlands, and Turkey.¹⁹

Recent Developments (2024-2025)

In 2024, Good Judgment significantly expanded its partnerships and media presence. The organization collaborated with major financial institutions including UBS Asset Management and Man Group on public forecasting challenges, while also hosting private challenges for organizations seeking to identify internal forecasting talent.²⁰ Superforecasters were featured in The Economist’s “The World Ahead 2025” issue with predictions on US tariffs, elections in Germany, Canada, and Australia, China’s inflation, and the Russia-Ukraine conflict.²¹

The company demonstrated continued forecasting superiority, with superforecasters proving 30% more accurate than futures markets on Federal Reserve, Bank of England, and European Central Bank interest rate decisions during 2024-2025.²² In 2025, Good Judgment won an Honorable Mention in the IF Awards from the Association of Professional Futurists for its work with UK partner ForgeFront on the Future.Ctrl methodology.²³ The organization also launched Advanced Judgment & Modeling as a next-level training program for graduates of its two-day workshops and began experimenting with hybrid AI-superforecasting integration.²⁴

Key People and Leadership

Philip Tetlock serves as co-founder and is the intellectual architect of the superforecasting approach.²⁵ A psychology professor at the University of Pennsylvania, Tetlock authored both Expert Political Judgment (which documented the limitations of expert forecasting) and Superforecasting: The Art and Science of Prediction (co-authored with Dan Gardner), which popularized the findings from the Good Judgment Project.²⁶ His research focuses on decision-making, expert judgment, and the cognitive characteristics that enable accurate probabilistic reasoning.

Barbara Mellers, also a co-founder, is a decision scientist at the University of Pennsylvania who co-led the Good Judgment Project with Tetlock.²⁷ Her research contributions focused on the psychological strategies that improve forecasting accuracy, including the role of cognitive ability, training interventions, and team collaboration in enhancing predictions.

Dr. Warren Hatch serves as CEO of Good Judgment Inc.²⁸ He holds a PhD from Oxford, previously worked on Wall Street at Morgan Stanley and a boutique firm, and is both a CFA Charterholder and a practicing superforecaster. Hatch has represented Good Judgment at high-profile events including the UK Department for Environment, Food and Rural Affairs (DEFRA) Futures Trend Briefing in November 2024 and the UN OCHA Global Humanitarian Policy Forum in December 2025.²⁹

Marc Koehler is Senior Vice President at Good Judgment Inc. and leads workshops on forecasting precision.³⁰ A former diplomat and practicing superforecaster, Koehler brings both domain expertise and forecasting skill to the organization’s training programs.

Michael Story, former Managing Director at Good Judgment, founded the Swift Centre to apply GJP methods commercially after his tenure with the organization.³¹ His background spans hedge funds, consulting, psychometrics, and risk quantification, reflecting the interdisciplinary nature of professional forecasting.

The superforecaster network itself represents a key organizational asset. These individuals span six continents and multiple languages, bringing decades of combined experience in probabilistic forecasting.³² Superforecasters are rigorously selected through performance on Good Judgment Open, with only the most consistently accurate forecasters qualifying for professional roles. Research indicates that superforecasters share certain characteristics including high cognitive ability, political knowledge, open-mindedness, pattern detection skills, and a deliberate practice mindset.³³

Methodology and Approach

Good Judgment’s forecasting methodology combines several key elements that distinguish it from traditional expert prediction. The approach emphasizes probabilistic thinking rather than binary yes/no predictions, with forecasters assigning percentage probabilities to different outcomes.³⁴ This allows for more nuanced assessment and enables scoring through Brier scores, which penalize both overconfidence and underconfidence.

The organization employs systematic bias training to help forecasters recognize and correct common cognitive errors. Research from the Good Judgment Project found that training on biases, combined with aggregation algorithms and team deliberation, significantly improved forecasting accuracy.³⁵ The project’s success in the IARPA tournament was partly attributed to these training interventions, which proved more effective than simply recruiting domain experts.

Team collaboration plays a crucial role in Good Judgment’s approach. During the ACE tournament, teams of 12 superforecasters consistently outperformed larger groups of regular forecasters, suggesting that high-skill aggregation matters more than crowd size.³⁶ Good Judgment typically assigns approximately 40 superforecasters to each client question, balancing the benefits of diverse perspectives with the premium placed on individual skill.³⁷

The organization also emphasizes long-term tracking and accountability. Forecasters make predictions on questions with clear resolution criteria and specified resolution dates, often ranging from months to years in the future.³⁸ This creates a track record that enables both individual skill assessment and continuous methodological improvement. Research analyzing five years of Good Judgment Project data found that compromise (averaged) forecasts from multiple forecasters were consistently more accurate than individual predictions and improved as events neared their resolution dates.³⁹

Performance and Track Record

Good Judgment’s empirical track record provides strong evidence for the effectiveness of its approach. During the IARPA ACE tournament (2011-2015), the Good Judgment Project achieved more than 50% improvement over control groups—the largest effect in the forecasting literature at that time.⁴⁰ Superforecasters demonstrated the ability to anticipate events 400 days in advance as accurately as other forecasters could predict them at 150 days out.⁴¹

Comparative performance metrics highlight the organization’s advantages across multiple benchmarks. Superforecasters outperformed US intelligence analysts with access to classified information by more than 30%, competing IARPA teams by 35-72%, and client experts combined with crowd-wisdom groups in policy forecasts.⁴² In head-to-head comparisons, 100 superforecasters defeated hybrid systems combining machine learning with 1,000+ crowd forecasts in 100% of test cases.⁴³

Recent performance demonstrates sustained accuracy. In 2023, Good Judgment superforecasters earned full marks on 8 out of 9 resolved forecasts in The Economist, correctly predicting global economic growth at 3%, China’s growth at 5%, and that Putin would not be ousted from power.⁴⁴ The organization also outperformed Financial Times readers (8,500 participants) on forecasts for 2023 events and proved 30% more accurate than futures markets on central bank interest rate decisions during 2024-2025.⁴⁵

As of September 2023, Good Judgment had resolved over 554 questions since 2015, with superforecasters placing the highest probability on the correct outcome across the majority of forecasting days.⁴⁶ The organization reports significantly lower Brier scores (indicating higher accuracy) compared to peer forecasters, with consistent performance maintained across diverse question types spanning geopolitics, economics, and social trends.⁴⁷

Funding and Operations

Good Judgment Inc. operates as a commercial entity with approximately 30 employees and annual revenue of approximately $5.4 million as of the most recent available data.⁴⁸ The company is headquartered at 100 Park Avenue, Floor 16, New York City.

The organization’s roots lie in government-funded research, specifically the IARPA-sponsored ACE tournament that ran from 2011 to 2015.⁴⁹ This research funding from the US intelligence community established the empirical foundation for Good Judgment’s commercial services. Since transitioning to a for-profit model, the company has diversified its revenue sources across government contracts, corporate clients in energy and finance sectors, and nonprofit organizations.⁵⁰

Recent grants demonstrate Good Judgment’s continued relevance to high-stakes decision-making in the effective altruism and global health communities. In March 2025, GiveWell paid Good Judgment Inc. $72,000 (50% of a $144,000 total project) to provide superforecaster predictions about US government foreign aid funding levels, with Open Philanthropy contributing the remaining 50%.⁵¹ This funding commissioned six forecasts on potential US foreign aid cuts, particularly focused on global health programs. Earlier contracts from Coefficient Giving included $150,000 for H5N1 forecasts (February 2023) and an unspecified amount for forecasting work on power-seeking AI (January 2023).⁵²

The company also receives support for collaborative projects. A Future Fund grant supported Good Judgment’s partnership with Metaculus on forecasting projects related to Our World In Data metrics, though the specific grant amount was not publicly disclosed.⁵³ These diverse funding sources reflect Good Judgment’s positioning at the intersection of commercial forecasting, academic research, and public-interest applications.

Products and Services

FutureFirst™ is Good Judgment’s flagship product, providing clients with continuous forecasting insights on strategic questions.⁵⁴ The platform operates 24/7 with approximately 40 superforecasters assigned per question, delivering probabilistic predictions on newsworthy and client-specific topics. The service reported 100% renewal rates among clients in Q4 2024, suggesting high satisfaction with the forecasting outputs.⁵⁵

Good Judgment Open serves as both a public forecasting platform and a recruitment pipeline for identifying new superforecasters.⁵⁶ The platform is free to use and functions similarly to golf par—users can benchmark their forecasting accuracy against questions with known resolutions. In 2024, GJ Open hosted public challenges for organizations including UBS Asset Management, Man Group, Fujitsu, City University of Hong Kong, Harvard Kennedy School, and The Economist.⁵⁷ The platform also supports private challenges for organizations seeking to identify internal forecasting talent and train staff in probabilistic thinking.

Training and workshops represent a growing component of Good Judgment’s business. The organization conducts forecasting training workshops, seminars, and presentations for hundreds of participants across government, nonprofit, and private sectors.⁵⁸ In 2025, Good Judgment launched an executive education program in Superforecasting Workshops for decision-makers, with clients including a major technology company, an oil multinational, and multiple investment funds.⁵⁹ The company also introduced Advanced Judgment & Modeling as a next-level training program for graduates of its two-day workshop.⁶⁰

Custom forecasting services allow clients to commission superforecaster predictions on proprietary strategic questions. Good Judgment frames client questions for optimal forecasting, assigns superforecaster teams, and delivers aggregated probabilistic predictions with documented track records. Clients span energy providers (for geopolitical and regulatory risk assessment), financial services, government agencies, and nonprofits.⁶¹

Partnerships and Collaborations

Good Judgment has established strategic partnerships across media, government, business, and research sectors. The organization maintains an 11-year collaboration with The Economist, with superforecasters regularly featured in the magazine’s annual outlook publications and participating in forecasting challenges on major global events.⁶² Coverage has also appeared in The New York Times, Wired, Vox, Financial Times, Bloomberg, Newsweek, The Guardian, and Forbes.⁶³

In the UK government sector, Good Judgment partners with ForgeFront and the UK Government’s Futures Procurement Network.⁶⁴ This collaboration included Good Judgment CEO Dr. Warren Hatch’s presentation at the Department for Environment, Food and Rural Affairs (DEFRA) Futures Trend Briefing in November 2024 on the role of superforecasting in biological security strategy. The partnership with ForgeFront earned an Honorable Mention in the 2025 IF Awards from the Association of Professional Futurists for their joint work on the Future.Ctrl methodology.⁶⁵

A significant research collaboration with Metaculus brought together two of the largest human judgment forecasting communities globally.⁶⁶ Supported by a Future Fund grant, the partnership involves cohorts of superforecasters from Good Judgment and Pro Forecasters from Metaculus making predictions on identical questions about technological advances, global development, and social progress across time horizons ranging from one to 100 years. This collaboration enables comparison of forecasting methodologies and results between the two leading platforms.

Good Judgment also works with educational institutions including Harvard Kennedy School and City University of Hong Kong, hosting forecasting challenges and tournaments.⁶⁷ The organization partnered with the Alliance for Decision Education on forecasting tournaments and hosted the 2025 University Forecasting Challenge, with winners receiving seats in the Superforecasting workshop running from August to December 2025.⁶⁸

Relationship to AI Safety and Effective Altruism

Good Judgment has limited direct involvement in AI safety research or existential risk mitigation, with the organization’s primary focus remaining on geopolitical and strategic forecasting.⁶⁹ However, several connections exist between Good Judgment’s work and the AI safety community.

The organization has received funding from effective altruism-aligned sources for forecasting projects with potential relevance to AI risk. Coefficient Giving funded a forecasting review on power-seeking AI in January 2023, though the specific amount and detailed findings were not publicly disclosed.⁷⁰ This represents one of the few documented instances of Good Judgment directly engaging with AI safety questions. The organization’s March 2025 collaboration with GiveWell and Open Philanthropy focused on US foreign aid rather than AI-specific risks, but demonstrates ongoing relationships with effective altruism funders.⁷¹

Within the effective altruism community, there has been discussion about the concept of “good judgment” as a critical trait for impactful decision-making, though without a standardized definition.⁷² Community members describe good judgment as mental processes leading to good decisions, comprising understanding of the world and effective heuristics. There is acknowledged overlap between EA’s concept of good judgment and LessWrong-style rationality, though some EA community members find the LessWrong approach off-putting.⁷³ Rebranding rationality insights as “good judgment” has been suggested as a way to bridge this divide.

The EA community has also debated whether good judgment and forecasting skill represent distinct capabilities.⁷⁴ Good Judgment’s track record—with superforecasters outperforming prediction markets by 15-30% and intelligence analysts by 25-30%—provides empirical evidence relevant to these discussions.⁷⁵ However, questions remain about whether forecasting accuracy on geopolitical questions translates to sound judgment on complex strategic questions like AI risk trajectories or optimal intervention strategies.

The potential application of Good Judgment’s methodology to AI safety forecasting remains largely unexplored. While the organization has demonstrated exceptional accuracy on questions with clear resolution criteria and relatively short time horizons (typically under 2 years), AI safety involves longer timeframes, deeper uncertainty, and questions that may be difficult to operationalize as verifiable predictions.⁷⁶ Some EA community discussions have explored the use of superforecasting for questions like “Is power-seeking AI an existential risk?”, though with mixed views on the appropriateness of forecasting for such fundamental strategic questions.⁷⁷

Criticisms and Limitations

Despite Good Judgment’s impressive track record, several criticisms and limitations have been identified in research and community discussions. A key methodological concern involves sample size effects on superforecaster identification. Research attempting to replicate the superforecasting hypothesis found that with a sample of only 195 participants and identification periods of less than a year, no superforecasters were identified.⁷⁸ This suggests the effect may be difficult to reproduce in smaller pools and raises questions about the statistical robustness of identifying truly exceptional forecasters versus observing random variation in performance.

Inherent cognitive biases remain a challenge even for skilled forecasters. The effectiveness of judgmental forecasting approaches is fundamentally constrained by forecasters’ inherent biases, which can lead to inadequate forecasts and failure to acknowledge poor performance.⁷⁹ While Good Judgment employs bias training, research indicates that the ability to engage in reflective thinking—interrogating initial gut feelings—appears to be partly innate and only partly developed through training.⁸⁰ This suggests limits to how much training can improve forecasting accuracy.

Domain limitations represent another significant concern. The Good Judgment Project focused mainly on geopolitical questions, which may not generalize to other domains.⁸¹ The organization explicitly acknowledges focusing on “low- or messy-data questions” where machine learning models show limited promise, according to researchers Tetlock and Koehler.⁸² This narrow focus means Good Judgment’s methods may not transfer effectively to domains with different characteristics, such as technological forecasting, scientific predictions, or questions involving rapid capability shifts.

Aggregation failures challenge the “wisdom of crowds” assumption underlying some forecasting approaches. Research found that when 45 people were asked to answer a factual question, the average was significantly inaccurate, though some individual forecasters performed well.⁸³ This highlights that aggregation methods matter considerably and simple averaging may not capture the value of superior forecasters. Good Judgment’s use of skilled superforecasters partially addresses this concern, but questions remain about optimal aggregation approaches.

Hybrid forecasting uncertainty clouds the comparison between human and machine learning approaches. Evidence on whether hybrid approaches combining human and machine learning forecasts actually outperform human-only forecasts remains unclear, with only anecdotal reports of better Brier scores from at least one participant.⁸⁴ As AI capabilities advance, the relative value of human superforecasters versus algorithmic approaches may shift, potentially undermining Good Judgment’s competitive advantages.

Long-term and existential questions pose particular challenges for Good Judgment’s methodology. The organization’s track record primarily involves questions with resolution timeframes under 2 years and clear verification criteria.⁸⁵ Questions about existential risks, transformative AI timelines, or century-scale social changes may be fundamentally different from the geopolitical questions where superforecasters have demonstrated accuracy. The applicability of Good Judgment’s methods to these higher-stakes, longer-horizon questions remains uncertain.

Key Uncertainties

Several important questions remain unresolved regarding Good Judgment’s methodology, performance, and potential applications:

Generalization across domains: Can superforecasting accuracy on geopolitical questions translate to other domains like technology forecasting, AI risk assessment, or scientific predictions? The organization’s focus on geopolitical and economic questions leaves open whether the same individuals and methods would perform comparably on fundamentally different question types.

Longevity of forecaster skill: Do superforecasters maintain their accuracy advantages over decades, or does performance degrade over time? While Good Judgment reports consistent performance from 2015 to present, the longest individual track records span only about 10-15 years, leaving uncertainty about multi-decade persistence of forecasting skill.

Scalability limits: As Good Judgment grows and employs more professional superforecasters, will the organization maintain its performance advantages? There may be limits to the supply of truly exceptional forecasters, and professionalization could introduce different incentives that affect forecasting accuracy.

AI-human hybrid approaches: How should superforecaster insights be combined with machine learning forecasts to achieve optimal accuracy? The organization is experimenting with hybrid AI-superforecasting integration, but optimal architectures and the long-term comparative advantage of human forecasters remain unclear.

Causal understanding versus prediction: Does accurate forecasting on geopolitical questions indicate deep causal understanding, or primarily pattern recognition and probabilistic reasoning? This distinction matters for whether superforecasters can provide valuable strategic guidance beyond point predictions.

Training effectiveness limits: How much can forecasting training improve accuracy for typical individuals, and what proportion of the population has the cognitive capacity to become highly skilled forecasters? Understanding these limits would inform decisions about investing in forecasting training versus other approaches to decision quality.

Optimal question framing: What characteristics of questions make them most suitable for superforecaster prediction versus other approaches like expert analysis, prediction markets, or algorithmic models? Good Judgment acknowledges focusing on “messy-data questions,” but the boundaries of this category remain imprecisely defined.