Back
Samotsvety Track Record
websamotsvety.org·samotsvety.org/track-record/
Samotsvety is a prominent forecasting group whose probability estimates on AI timelines and risks are frequently cited in AI safety discussions; this page documents their competitive track record, providing context for assessing the credibility of their forecasts.
Metadata
Importance: 38/100organizational reportreference
Summary
Documents the track record and performance history of Samotsvety, a high-performing forecasting team that has achieved top rankings in prediction competitions including INFER and Good Judgment Open. Provides evidence of the team's calibration and predictive accuracy across diverse domains, including AI and existential risk questions.
Key Points
- •Samotsvety has consistently ranked among top performers in major forecasting tournaments including INFER and Good Judgment Open
- •Individual team members have demonstrated exceptional predictive capabilities in competitive forecasting environments
- •The track record provides credibility for Samotsvety's forecasts on AI timelines, existential risk, and policy-relevant questions
- •Demonstrated calibration across competitions supports use of Samotsvety forecasts as reference points in AI safety discourse
- •The team's performance history is relevant context for evaluating their published probability estimates on AI safety topics
Review
The Samotsvety Forecasting team represents a remarkable collective of probabilistic reasoning experts who have distinguished themselves through consistently superior forecasting performance. Their track record spans multiple platforms like INFER, Good Judgment Open, and Metaculus, where they've repeatedly demonstrated an ability to make highly accurate predictions across diverse domains including geopolitics, technology, and global events.
While their achievements are impressive, the document primarily serves as a track record compilation rather than a detailed methodological exposition. The team's success appears rooted in individual members' analytical skills, domain expertise, and refined probabilistic reasoning techniques. Their performance is quantified through Brier scores, with team members consistently scoring well below median predictions, indicating significantly more accurate forecasting. The inclusion of several Superforecasters™ and individuals with diverse academic backgrounds suggests that their success stems from a combination of interdisciplinary knowledge, rigorous analytical approaches, and a nuanced understanding of uncertainty.
Cached Content Preview
HTTP 200Fetched Apr 11, 20266 KB
Team achievements
The Samotsvety Forecasting team at INFER/CSET-Foretell—composed out of Nuño, Misha, Eli—took:
1st place in 2020 , with a relative score of -0.912 (vs. -0.062 by the 2nd best team). Individually we finished 5th, 6th, and 7th .
1st place in 2021 with a relative score of -3.259 (vs. -0.889 by the 2nd best team and vs. -0.267 by “Pro Forecasters”). Individually we finished 1st, 2nd, 4th and 5th .
We still hold 1st, 2nd, 3rd, 4th places in INFER’s all time ranking.
We still hold 1st place in 2022 , despite us reducing our participation.
A few of our forecasters have now become Superforecasters™, indicated below.
The above links require signing in, so here is a screenshot of our performance in the 2021 season:
Note that we are a bit more than twice as good as the next best team, in terms of the relative Brier score
As of 2022-09-15, we are also 4th on the nascent Insight Prediction leaderboard as a result of one (1) large bet when we correctly foresaw the Russian invasion of Ukraine and put our money where our mouth was:
Individuals
Individually, Samotsvety members are, as of 2022-09-15, occupying all top 4 spots on INFER’s overall leaderboard :
For each forecaster below, the time at which their profile was last updated is indicated in a superscript.
Misha Yagudin 2023-05-27
Runs a research consultancy Arb Research
GJOpen Brier score 0.185 vs. 0.275 median, ratio 0.67
7th and 3rd place forecaster in the first two seasons of CSET-Foretell/INFER ( I , II ); 3rd all-time (as of 2022/9/15).
Nuño Sempere 2023-05-27
Indepedent consultant at Shapley Maximizers ÖU , previously a researcher at the Quantified Uncertainty Research Institute.
GJOpen Brier score 0.206 vs. 0.29 median, ratio 0.71
Top 5 forecaster in the first season of INFER, 2nd best in the second season ; 2nd all-time (as of 2024/02/03)
Alex Lyzhov 2023-05-27
ML PhD student at NYU
Participates primarily in AI-related forecasts due to expertise
Active Metaculus contributor
Eli Lifland 2023-05-27
Figuring out what’s up with AI alignment (and sometimes other stuff), writing Foxy Scout .
GJOpen Brier score 0.23 vs. 0.301 median, ratio 0.76
1st forecaster of all time on CSET-Foretell/INFER, as well as for seasons one and two individually (as of 2024/02/04)
2nd in Metaculus Economist 2021 tournament , 1st in Salk Tournament (as of 2022/9/10).
Track record is described in detail here
Jonathan Mann 2023-05-27
Professional in the financial sector
GJOpen Brier score 0.156 vs. 0.246 median, ratio 0.63
Pro forecaster on INFER
About + Github
Superforecaster™
INFER All-Star
Juan Cambeiro 2023-05-27
MPH student in epidemiology at Columbia University
Biosecurity fellow at the Institute for Progress
Good Judgment Superforecaster
Analyst for Metaculus
GJOpen Brier score 0.25 vs. 0.317 median, ratio 0.79
First in IARPA’s FOCUS Tour
... (truncated, 6 KB total)Resource ID:
c7b435dfad2f7ca2 | Stable ID: sid_JArvEbno2v