Back
Calibration City - Prediction Market Calibration & Accuracy Analysis
webmanifund.org·manifund.org/projects/calibration-city
Calibration City is a forecasting calibration analysis tool aggregating data from major prediction markets (Kalshi, Manifold, Metaculus, Polymarket). While not directly AI safety research, calibration and forecasting accuracy are relevant to AI safety through improving epistemic tools used by the community for risk assessment.
Metadata
Importance: 28/100tool pagetool
Summary
Calibration City is a web platform that aggregates and analyzes prediction market data from Kalshi, Manifold, Metaculus, and Polymarket, covering over 130,000 markets. It provides calibration plots and accuracy visualizations allowing users to compare platform performance, filter by market characteristics, and understand how factors like trade volume and market duration affect forecasting accuracy. The project received $3,500 from the Manifold Community Fund and additional Manifund grant support.
Key Points
- •Aggregates 130,000+ markets from Kalshi, Manifold, Metaculus, and Polymarket for calibration and accuracy analysis
- •Provides interactive calibration plots and accuracy visualizations with filtering by keyword, category, duration, volume, and more
- •Allows comparison of platform-level overconfidence/underconfidence and how factors like trade volume affect accuracy
- •Includes beginner-friendly Socratic introduction to forecasting concepts alongside advanced analytical tools
- •Awarded $3,500 from Manifold Community Fund and featured in Nuño Sempere's forecasting newsletter
Cited by 1 page
| Page | Type | Quality |
|---|---|---|
| Manifold (Prediction Market) | Organization | 43.0 |
Cached Content Preview
HTTP 200Fetched Apr 11, 202619 KB
Calibration City | Manifund
Dec
JAN
Feb
15
2025
2026
2027
success
fail
About this capture
COLLECTED BY
Collection: Common Crawl
Web crawl data from Common Crawl.
TIMESTAMPS
The Wayback Machine - https://web.archive.org/web/20260115141822/https://manifund.org/projects/calibration-city
Manifund
Home
Login
About
People
Categories
Newsletter
Home
About
People
Categories
Login
Create
6
Calibration City
EA Community Choice
Forecasting
wasabipesto
Complete
Grant
$8,864raised
p]:prose-li:my-0 text-gray-900 prose-blockquote:text-gray-600 prose-a:font-light prose-blockquote:font-light font-light break-anywhere empty:prose-p:after:content-["\00a0"]">
Project summary
I’ve been working on Calibration City, a site for prediction market calibration and accuracy analysis. I want the site to be useful for experienced prediction market users as well for as people who have never heard of them before.
Example user questions we aim to answer include:
I'm interested in sports, how good is Manifold at predicting games a week in advance? Do other sites have a better track record?
This PredictIt market is trading at 90¢ but has less than 2000 shares in volume. How often does a market like that end up being wrong?
I’m worried about the accuracy of markets that won’t resolve for a long time. What is the typical accuracy of a market over a year away from resolution?
What have you done so far?
Calibration City is currently live! We completed the MVP in January 2024 with additional features landing in February and March. We integrate data from Kalshi, Manifold, Metaculus, and Polymarket, with over 130,000 total markets and over 300 visitors in the past month.
There are currently two main visualizations: calibration and accuracy. The calibration page shows a standard calibration plot for each supported platform. The user can choose how markets are sorted into bins along the x-axis (by the market probability at a specific point, or a time-weighted average). They can also apply weighting to each market based on values such as the market volume, length, or number of traders. Users can filter the total set of markets used for analysis based on keyword, category, duration, volume, or other features. Is Polymarket consistently overconfident? Underconfident? What about on long-term markets?
The accuracy plot allows users to directly compare different factors’ effects on market accuracy. In addition to the standard filters and binning options, the user can select a factor such as the market date, total trade volume, market length, or number of traders. With this additional axis, users can learn how (or if) those factors actually impact market accuracy. Does higher trade volume really increase accuracy? If so, by how much? What about more recent markets?
The beg
... (truncated, 19 KB total)Resource ID:
bfde7f675f3f5403 | Stable ID: sid_h2F4k0CXpO