ForecastBench: Dynamic LLM Forecasting Benchmark

Summary

ForecastBench is a contamination-free benchmark that evaluates LLM forecasting accuracy against human comparison groups, including superforecasters. It maintains both a baseline leaderboard (no tools) and a tournament leaderboard (with scaffolding/tools), and projects when LLMs will reach superforecaster-level performance.

Key Points

• •Dynamic, contamination-free benchmark preventing LLMs from training on benchmark questions, ensuring valid capability measurement.
• •Compares LLM forecasting performance against human baselines including superforecasters as a proxy for general intelligence.
• •Dual leaderboards: baseline (raw model performance) and tournament (with tool use, fine-tuning, ensembling).
• •Tracks historical progress in LLM forecasting capabilities and projects date of LLM-superforecaster parity.
• •Open to public submissions, enabling broad participation in capability evaluation.