The Price of Progress: Algorithmic Efficiency and the Falling Cost of AI Inference

The Price of Progress Algorithmic Efficiency and the Falling Cost of AI Inference The Price of Progress Algorithmic Efficiency and the Falling Cost of AI Inference Hans Gundlach MIT CSAIL, MIT FutureTech hansgund@mit.edu &Jayson Lynch MIT CSAIL, MIT FutureTech jaysonl@mit.edu &Matthias Mertens MIT Sloan, MIT FutureTech mmertens@mit.edu &Neil Thompson 1 1 footnotemark: 1 MIT CSAIL, MIT FutureTech neil_t@mit.edu Corresponding authors. hansgund@mit.edu, neil_t@mit.edu Abstract Language models have seen enormous progress on advanced benchmarks in recent years, but much of this progress has only been possible by using more costly models. Benchmarks may therefore present a warped picture of progress in practical capabilities per dollar . To remedy this, we use data from Artificial Analysis and Epoch AI to form the largest dataset of current and historical prices to run benchmarks to date. We find that the price for a given level of benchmark performance has decreased remarkably fast, around 5 × 5\times to 10 × 10\times per year, for frontier models on knowledge, reasoning, math, and software engineering benchmarks. These reductions in the cost of AI inference are due to economic forces, hardware efficiency improvements, and algorithmic efficiency improvements. Isolating out open models to control for competition effects and dividing by hardware price declines, we estimate that algorithmic efficiency progress is around 3 × 3\times per year. Finally, we recommend that evaluators both publicize and take into account the price of benchmarking as an essential part of measuring the real-world impact of AI. 1 1 1 This paper was accepted to the NeurIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, and Scaling ( https://sites.google.com/view/llm-eval-workshop ). 1 Introduction A critical dimension of the real-world impact of language models (and AI systems in general) is cost, which is often ignored in discourses around evaluations and AI performance. Popular blog posts have ignited discussions on potentially large declines in prices for accessing a given level of (high) LLM performance ( appenzeller2024llmflation ) . At the same time, epoch2025llminferencepricetrends finds that, controlling for benchmark performance, LLM token prices may be decreasing by factors of 10–1,000 × \times per year, depending on the performance level. On the other hand, erol2025cost finds that the cost-of-pass on benchmarks like MATH 500 and AIME 2024 has gone down by 24.5 × \times and 3.23 × \times per year, respectively. Understanding these price trends is key for many issues, such as predicting the cost efficiency of models versus labor-based work, or democratizing access to state-of-the-art AI capabilities. 2 2 2 Documenting changes in quality-adjusted AI prices is also relevant to economic science, as these data can be helpful for understanding substitution elasticities between labor and AI inputs. In this study, we carefully examine how

... (truncated, 80 KB total)