comprehensive study on agent security
paperAuthors
Credibility Rating
Good quality. Reputable source with community review or editorial standards, but less rigorous than peer-reviewed venues.
Rating inherited from publication venue: arXiv
A 2024 UC Davis paper systematically cataloging security vulnerabilities in LLM-based AI agents, relevant to researchers and practitioners deploying agentic systems in real-world environments.
Paper Details
Metadata
Abstract
AI agents have been boosted by large language models. AI agents can function as intelligent assistants and complete tasks on behalf of their users with access to tools and the ability to execute commands in their environments. Through studying and experiencing the workflow of typical AI agents, we have raised several concerns regarding their security. These potential vulnerabilities are not addressed by the frameworks used to build the agents, nor by research aimed at improving the agents. In this paper, we identify and describe these vulnerabilities in detail from a system security perspective, emphasizing their causes and severe effects. Furthermore, we introduce defense mechanisms corresponding to each vulnerability with design and experiments to evaluate their viability. Altogether, this paper contextualizes the security issues in the current development of AI agents and delineates methods to make AI agents safer and more reliable.
Summary
This paper provides a comprehensive security analysis of LLM-based AI agents, identifying critical vulnerabilities arising from their tool-use and command-execution capabilities. The authors examine these vulnerabilities from a system security perspective and propose corresponding defense mechanisms, evaluating their effectiveness experimentally.
Key Points
- •Identifies and categorizes novel security vulnerabilities unique to LLM-based agents that existing frameworks and research do not adequately address.
- •Analyzes the full agent development workflow from a system security lens, detailing root causes and potential impacts of each vulnerability.
- •Proposes and experimentally evaluates defense mechanisms tailored to each identified vulnerability.
- •Highlights that agent tool-use and environment interaction create attack surfaces absent in standalone LLMs, including prompt injection and command execution risks.
- •Contextualizes AI agent security within the broader landscape of AI safety, bridging cybersecurity and AI research communities.
Cited by 1 page
| Page | Type | Quality |
|---|---|---|
| Tool Use and Computer Use | Capability | 67.0 |
Cached Content Preview
Security of AI Agents
\NewDocumentCommand \code
v #1
\addbibresource main.bib
Security of AI Agents
Yifeng He
UC Davis
Davis, USA
yfhe@ucdavis.edu
Ethan Wang
UC Davis
Davis, USA
ebwang@ucdavis.edu
Yuyang Rong
UC Davis
Davis, USA
PeterRong96@gmail.com
Zifei Cheng
UC Davis
Davis, USA
zfcheng@ucdavis.edu
Hao Chen
UC Davis
Davis, USA
chen@ucdavis.edu
Abstract
The study and development of AI agents have been boosted by large language models.
AI agents can function as intelligent assistants and complete tasks on behalf of their users
with access to tools and the ability to execute commands in their environments,
Through studying and experiencing the workflow of typical AI agents,
we have raised several concerns regarding their security.
These potential vulnerabilities are not addressed by the frameworks used to build the agents,
nor by research aimed at improving the agents.
In this paper, we identify and describe these vulnerabilities in detail from a system security perspective,
emphasizing their causes and severe effects.
Furthermore, we introduce defense mechanisms corresponding to each vulnerability with meticulous design and experiments to evaluate their viability.
Altogether, this paper contextualizes the security issues in the current development of AI agents
and delineates methods to make AI agents safer and more reliable.
1 Introduction
AI agents are robots in cyberspace, executing tasks on behalf of their users.
To understand their user’s command,
they send the input prompts as requests to foundation AI models, such as large language models (LLMs).
The responses generated by the model may contain the final actions of the agent or further instructions.
To execute the actions , the agent invokes tools ,
which may run local computations or send requests to remote hosts, such as querying search engines.
The tools output results and feedback to the AI model for the next round of actions.
By invoking tools, AI agents are granted the ability to interact with the real world.
Since AI agents depend on their AI model to understand user input and environment feedback
and generate actions to use tools, we say that the AI model is the backbone of the agent.
We summarize the basic architecture of LLM-based AI agents in Figure 1 .
Traditional agents operate on pre-defined rules [ castelfranchi1998modelling , wilkins2014practical ] or
reinforcement learning models [ isbell2001social ] ,
making them hard to generalize to new tasks and different tools.
LLM-based AI agents, on the contrary,
can be practical in various tasks benefiting from enormous pre-training knowledge
and the ability to read tool documentation as additional prompts.
We use the term AI agent to denote all LLM-based agents in this paper.
Over the years, AI agents have showcased their outstanding performance on tasks including but not limited to
writ
... (truncated, 64 KB total)3aec04f6fbc348bf | Stable ID: sid_NsmkuFzLnf