Scaling Up AI: Trends, Capabilities, and Implications — Our World in Data

Scaling up: how increasing inputs has made artificial intelligence more capable - Our World in Data Home Artificial Intelligence For most of Artificial Intelligence’s (AI’s) history, many researchers expected that building truly capable systems would need a long series of scientific breakthroughs: revolutionary algorithms, deep insights into human cognition, or fundamental advances in our understanding of the brain. While scientific advances have played a role, recent AI progress has revealed an unexpected insight: a lot of the recent improvement in AI capabilities has come simply from scaling up existing AI systems. 1 

 Here, scaling means deploying more computational power, using larger datasets, and building bigger models. This approach has worked surprisingly well so far. 2 Just a few years ago, state-of-the-art AI systems struggled with basic tasks like counting. 3 4 Today, they can solve complex math problems, write software, create extremely realistic images and videos, and discuss academic topics. 

 This article will provide a brief overview of scaling in AI over the past years. The data comes from Epoch , an organization that analyzes trends in computing, data, and investments to understand where AI might be headed. 5 Epoch maintains the most extensive dataset on AI models and regularly publishes key figures on AI growth and change. 

 What is scaling in AI models? 

 Let’s briefly break down what scaling means in AI. Scaling is about increasing three main things during training, which typically need to grow together: 

 The amount of data used for training the AI; 
 The model’s size, measured in “parameters”; 
 Computational resources, often called "compute" in AI. 
 The idea is simple but powerful: bigger AI systems, trained on more data and using more computational resources, tend to perform better . Even without substantial changes to the algorithms, this approach often leads to better performance across many tasks. 6 

 Here is another reason why this is important: as researchers scale up these AI systems, they not only improve in the tasks they were trained on but can sometimes lead them to develop new abilities that they did not have on a smaller scale. 7 For example, language models initially struggled with simple arithmetic tests like three-digit addition, but larger models could handle these easily once they reached a certain size. 8 The transition wasn't a smooth, incremental improvement but a more abrupt leap in capabilities. 

 This abrupt jump in capability, rather than steady improvement, can be concerning. If, for example, models suddenly develop unexpected and potentially harmful behaviors simply as a result of getting bigger, it would be harder to anticipate and control. 

 This makes tracking these metrics important. 

 What are the three components of scaling up AI models? 

 Data: scaling up the training data 

 One way to view today's AI models is by looking at them as very sophisticated pattern recognit

... (truncated, 15 KB total)