Hoffmann et al. (2022)
paperAuthors
Credibility Rating
Good quality. Reputable source with community review or editorial standards, but less rigorous than peer-reviewed venues.
Rating inherited from publication venue: arXiv
Influential empirical study on compute-optimal scaling laws for language models, demonstrating that current large models are undertrained and establishing guidelines for efficient allocation of compute budgets across model size and training data.
Paper Details
Metadata
Abstract
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. We test this hypothesis by training a predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4$\times$ more more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks. This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage. As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher.
Summary
Hoffmann et al. (2022) investigates the optimal allocation of compute budgets between model size and training data for transformer language models. Through extensive experiments training over 400 models ranging from 70M to 16B parameters, the authors find that current large language models are significantly undertrained due to emphasis on model scaling without proportional increases in training data. They propose that compute-optimal training requires equal scaling of model size and training tokens—doubling model size should be accompanied by doubling training data. The authors validate this finding with Chinchilla (70B parameters), which matches Gopher's compute budget but uses 4× more data, achieving superior performance across downstream tasks and reaching 67.5% on MMLU, a 7% improvement over Gopher.
Cited by 6 pages
| Page | Type | Quality |
|---|---|---|
| Large Language Models | Capability | 60.0 |
| Large Language Models | Concept | 62.0 |
| AI Accident Risk Cruxes | Crux | 67.0 |
| Power-Seeking Emergence Conditions Model | Analysis | 63.0 |
| AI Scaling Laws | Concept | 92.0 |
| AI Proliferation | Risk | 60.0 |
Cached Content Preview
[2203.15556] Training Compute-Optimal Large Language Models
\correspondingauthor
email@email
\reportnumber 001
\correspondingauthor {jordanhoffmann|sborgeaud|amensch|sifre}@deepmind.com
Training Compute-Optimal Large Language Models
Jordan Hoffmann
Equal contributions
Sebastian Borgeaud
Equal contributions
Arthur Mensch
Equal contributions
Elena Buchatskaya
Trevor Cai
Eliza Rutherford
Diego de Las Casas
Lisa Anne Hendricks
Johannes Welbl
Aidan Clark
Tom Hennigan
Eric Noland
Katie Millican
George van den Driessche
Bogdan Damoc
Aurelia Guy
Simon Osindero
Karen Simonyan
Erich Elsen
Jack W. Rae
Oriol Vinyals
Laurent Sifre
Equal contributions
Abstract
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget.
We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant.
By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled.
We test this hypothesis by training a predicted compute-optimal model, Chinchilla , that uses the same compute budget as Gopher but with 70B parameters and 4 × \times more more data.
Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks.
This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage.
As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher .
1 Introduction
Recently a series of Large Language Models (LLMs) have been introduced (Brown et al., 2020 ; Lieber et al., 2021 ; Rae et al., 2021 ; Smith et al., 2022 ; Thoppilan et al., 2022 ) , with the largest dense language models now having over 500 billion parameters.
These large autoregressive transformers (Vaswani et al., 2017 ) have demonstrated impressive performance on many tasks using a variety of evaluation protocols such as zero-shot, few-shot, and fine-tuning.
The compute and energy cost for training large language models is substantial (Rae et al., 2021 ; Thoppilan et al., 2022 ) and rises with increasing model size.
In practice, the allocated training compute budget is often known in advance: how many accelerators are available and for how long w
... (truncated, 98 KB total)46fd66187ec3e6ae | Stable ID: sid_gOVKhh9R4G