2025 technical report
paperAuthors
Credibility Rating
Good quality. Reputable source with community review or editorial standards, but less rigorous than peer-reviewed venues.
Rating inherited from publication venue: arXiv
A landmark 2025 technical report providing the most systematic treatment to date of safety risks specific to multi-agent AI systems; essential reading for anyone working on agentic AI safety, governance of AI ecosystems, or cooperative AI research.
Paper Details
Metadata
Abstract
The rapid development of advanced AI agents and the imminent deployment of many instances of these agents will give rise to multi-agent systems of unprecedented complexity. These systems pose novel and under-explored risks. In this report, we provide a structured taxonomy of these risks by identifying three key failure modes (miscoordination, conflict, and collusion) based on agents' incentives, as well as seven key risk factors (information asymmetries, network effects, selection pressures, destabilising dynamics, commitment problems, emergent agency, and multi-agent security) that can underpin them. We highlight several important instances of each risk, as well as promising directions to help mitigate them. By anchoring our analysis in a range of real-world examples and experimental evidence, we illustrate the distinct challenges posed by multi-agent systems and their implications for the safety, governance, and ethics of advanced AI.
Summary
A comprehensive technical report from the Cooperative AI Foundation that taxonomizes risks in multi-agent AI systems, identifying three core failure modes (miscoordination, conflict, and collusion) and seven underlying risk factors. The authors ground their analysis in real-world examples and experimental evidence, arguing these risks are qualitatively distinct from single-agent safety challenges and require novel mitigation strategies spanning technical, governance, and ethical dimensions.
Key Points
- •Identifies three primary failure modes in multi-agent systems: miscoordination (agents failing to align on beneficial outcomes), conflict (competing objectives), and collusion (agents cooperating against human interests).
- •Enumerates seven systemic risk factors including incentive structures, information asymmetries, and network effects that can trigger or amplify multi-agent failures.
- •Argues multi-agent risks are qualitatively novel and not reducible to single-agent safety problems, requiring dedicated research and governance frameworks.
- •Draws on game theory, mechanism design, and multi-agent reinforcement learning to ground recommendations in established formal frameworks.
- •Produced collaboratively by researchers from Cooperative AI Foundation, Google DeepMind, Anthropic, CMU, Oxford, and other leading institutions, lending broad credibility.
Cited by 3 pages
| Page | Type | Quality |
|---|---|---|
| Collective Intelligence / Coordination | Capability | 56.0 |
| Cooperative AI | Approach | 55.0 |
| Multi-Agent Safety | Approach | 68.0 |
Cached Content Preview
[2502.14143] Multi-Agent Risks from Advanced AI
Multi-Agent Risks from Advanced AI
Lewis Hammond and Alan Chan and Jesse Clifton and Jason Hoelscher-Obermaier and Akbir Khan and Euan McLean and Chandler Smith and Wolfram Barfuss and Jakob Foerster and Tomáš Gavenčiak and The Anh Han and Edward Hughes and Vojtěch Kovařík and Jan Kulveit and Joel Z. Leibo and Caspar Oesterheld and Christian Schroeder de Witt and Nisarg Shah and Michael Wellman and Paolo Bova and Theodor Cimpeanu and Carson Ezell and Quentin Feuillade-Montixi and Matija Franklin and Esben Kran and Igor Krawczuk and Max Lamparth and Niklas Lauffer and Alexander Meinke and Sumeet Motwani and Anka Reuel and Vincent Conitzer and Michael Dennis and Iason Gabriel and Adam Gleave and Gillian Hadfield and Nika Haghtalab and Atoosa Kasirzadeh and Sébastien Krier and Kate Larson and Joel Lehman and David C. Parkes and Georgios Piliouras and Iyad Rahwan
(February 19 2025)
See covers/front.pdf
Multi-Agent Risks from Advanced AI
Lewis Hammond 1,2, * * *
Correspondence to lewis.hammond@cooperativeai.org .
Suggested citation: “Hammond et al. (2025). Multi-Agent Risks from Advanced AI . Cooperative AI Foundation, Technical Report #1.”
Author clusters are ordered by approximate magnitude of contribution and represent the lead author, organisers, major contributors, minor contributors, and advisors, respectively.
Within clusters, authors are listed alphabetically.
Full details of author roles are available in Appendix A .
Affiliations in parentheses indicate that the author’s work on this report was primarily completed while under that affiliation.
Due to the length of the author list, authorship does not entail endorsement of all claims in the report, nor does inclusion entail an endorsement on the part of any individual’s organisation.
In particular, contributions to this report reflect the views of the respective contributors and not necessarily the views of the Cooperative AI Foundation, its trustees, or funders.
Alan Chan 3,4 ,
Jesse Clifton 5,1 ,
Jason Hoelscher-Obermaier 6 ,
Akbir Khan 7,8,(1) ,
Euan McLean † ,
Chandler Smith 1
Wolfram Barfuss 9 ,
Jakob Foerster 2,10 ,
Tomáš Gavenčiak 11 ,
The Anh Han 12 ,
Edward Hughes 13 ,
Vojtěch Kovařík 14,(15) ,
Jan Kulveit 11 ,
Joel Z. Leibo 13 ,
Caspar Oesterheld 15 ,
Christian Schroeder de Witt 2 ,
Nisarg Shah 16 ,
Michael Wellman 17
Paolo Bova 12 ,
Theodor Cimpeanu 18,(19)
Carson Ezell 20 ,
Quentin Feuillade- Montixi 21,(†) ,
Matija Franklin 8 ,
Esben Kran 6 ,
Igor Krawczuk †,(22) ,
Max Lamparth 23 ,
Niklas Lauffer 24 ,
Alexander Meinke 25,(†) ,
Sumeet Motwani 2,(24) ,
Anka Reuel 23,20
Vincent Conitzer 15 ,
Michael Dennis 13 ,
Iason Gabriel 13 ,
Adam Gleave 26 ,
Gillian Hadfield 27 ,
Nika Haghtalab 24 ,
Atoosa Kasirzadeh 15 ,
Sébastien Krier 13 ,
Kate Larson 28,13 ,
Joel Lehman † ,
David C. Parkes 20 ,
Georgios Piliouras 13,29 ,
Iyad Rahwan
... (truncated, 98 KB total)772b3b663b35a67f | Stable ID: sid_tWCXEkTo6C