AI security: how to protect AI agents and applications

The adoption of Artificial Intelligence in business environments is changing the attack surface of applications. A model that only generates text presents a different level of exposure compared to an agent capable of querying internal documents, accessing a CRM, using APIs, or executing actions on other systems.

This evolution means that AI Security must address risks that do not appear in the same way in a traditional application. An agent can receive instructions from the user, but it can also interpret information from web pages, emails, documents, knowledge bases, and other external sources. At the same time, it may have tools and permissions that turn an incorrect response into an action with real consequences.

This exposure increases even more in Multi-Agent Systems, where different agents can operate with different tools, permissions, and responsibilities.

Protecting these systems requires a joint analysis of the model, the data it can access, the available tools, the identity used to operate, and the actions it is authorized to execute. Security shifts from focusing solely on what AI can generate to also depending on what it can do.

What is AI security and what changes when AI goes into production

AI Security encompasses the controls used to protect models, applications, and Artificial Intelligence agents from manipulation, unauthorized access to information, tool abuse, and unauthorized actions.

Many of these systems still rely on traditional cybersecurity principles. Authentication, authorization, data protection, identity management, segmentation, logging, and monitoring remain necessary. What changes is how the model interacts with these controls.

An AI agent can dynamically decide which tool to use during a workflow. It can also combine information obtained from different sources and use the result of one action to decide the next.

OpenAI precisely differentiates agents from simpler applications based on LLM by their ability to manage the execution of workflows and use tools to obtain information or perform actions on external systems. This autonomy introduces useful capabilities but also increases the importance of defining clear boundaries for their behavior.

The risk surface can grow as the system incorporates new capabilities. A chatbot without access to internal systems has a limited scope. If that same system can query confidential information, send messages, modify records, or execute operations via APIs, manipulation of the model can have broader consequences.

Prompt injection and manipulation of AI agents

Prompt injection attempts to modify the behavior of an AI application through instructions that conflict with the original user's intent or the established rules for the system.

The risk is particularly relevant in agents because potentially malicious instructions do not necessarily have to come from the user.

An agent researching information on the Internet may find instructions within a page. Another processing corporate documents may receive them from a file. A system connected to email may interpret content included in a message.

OpenAI points out that agents capable of navigating, retrieving information, and executing actions introduce new opportunities for this type of manipulation. The more sophisticated variants may resemble social engineering techniques: the content attempts to convince the model that a particular instruction is legitimate within the context of the task.

Direct prompt injection in AI applications

In a direct prompt injection, the manipulation comes directly from the user's interaction with the application.

The attacker may try to make the model ignore certain restrictions, reveal information that should remain protected, or use its capabilities outside the defined purpose for the system.

In an application that only generates content, the impact may focus on the produced response. When the same model controls an agent with tools, the situation changes because the instruction may attempt to influence subsequent decisions in the workflow.

For this reason, protection cannot rely solely on the model recognizing a malicious instruction. It must also limit what consequences a manipulation that manages to influence its behavior could have.

Indirect prompt injection in agents connected to external data

Indirect prompt injection introduces a different problem. The malicious instruction may be found within the content that the agent consults to perform a legitimate task.

Imagine an agent tasked with analyzing documents and preparing a report. One of those files could contain text designed to modify its behavior. Something similar can happen with web pages, emails, or information obtained from other applications.

The user does not need to have directly provided the problematic instruction. The workflow itself leads the agent to it.

This makes the source of information an important dimension of AI Security. Data retrieved from an external source may be necessary to complete a task, but it should not automatically acquire the same authority as the instructions that define the agent's behavior.

OpenAI recommends limiting agents' access to the information necessary to complete each task and carefully reviewing important actions before confirming them. These measures reduce the potential impact when an agent encounters manipulated content during its execution.

Data leakage and information exposure in AI systems

Data exposure does not solely depend on a model reproducing confidential information in a response.

An agent can retrieve information from different systems, incorporate it into its context, and subsequently use a tool to transmit it to another destination. The combination of data access + action capability modifies the potential scope of a leak.

For example, an agent may need to consult customer information to complete an internal process. That access does not mean it should be able to transmit all that data through any available tool.

The architecture must determine what information each agent can consult, how long it can use it, and what operations it can perform with it.

It is also relevant to control the information that is incorporated into memory, logs, or observability systems. A mechanism designed to improve traceability can become another exposure surface if it stores sensitive information without adequate controls.

Microsoft includes the data entered into agents' memory and the information transferred to tools among the responsibilities that must be protected during the deployment of agentic systems.

Therefore, data protection in AI Security requires establishing boundaries around the entire information journey: origin, retrieval, context, memory, tool, and destination.

Tool access and permissions for AI agents

Tools are one of the elements that turn a model into a system capable of acting.

OpenAI classifies agents' tools into capabilities related to data, actions, and orchestration. Some allow querying information; others can modify records, send communications, or execute operations on external systems.

From a security perspective, these capabilities should not be treated the same way.

An agent that can check the status of an order presents a different risk than one that can cancel it. Reading a record is not the same as modifying it. Preparing an email does not have the same consequences as automatically sending it.

The security of tools must consider both the available function and the permissions with which it is executed.

Least privilege applied to AI agents

The principle of least privilege states that each agent should only have the permissions necessary to fulfill its function.

Microsoft presents this principle as a design requirement for agentic systems: identity, scope, tool access, and auditability should be defined before expanding the agent's autonomy. It also recommends using specific identities and avoiding shared credentials that make it difficult to attribute an action later.

Applied to an AI agent, this means controlling four related dimensions: what data it can query → what tools it can use → what actions it can execute → on what resources it can do so.

An agent tasked with querying information should not receive write permissions simply because the integration used also offers that capability. Similarly, a tool incorporated during a test should be removed when it is no longer needed.

The goal is to reduce the potential impact of incorrect behavior or manipulation. If the agent does not have a certain capability, a malicious instruction cannot use it directly either.

Reversible and irreversible actions of agents

Not all tools require the same level of control.

A query can be repeated or discarded without modifying the state of a system. In contrast, deleting information, making a payment, changing permissions, deploying software, or sending data to a third party can produce difficult-to-reverse consequences.

OpenAI suggests evaluating tools by considering factors such as read-only versus write access, reversibility, required permissions, and financial impact. Higher-risk actions may use additional controls or be escalated to a person before execution.

This classification allows adapting controls to the potential impact.

An agent can automatically execute low-risk operations while maintaining an approval point for those that modify sensitive information, affect productive systems, or produce economic consequences.

Rogue actions and excessive agency in AI agents

An agent does not need to be fully compromised to execute a problematic action.

It can misinterpret an instruction, receive manipulated information, or select an inappropriate tool. The impact largely depends on the capabilities it has available.

OWASP uses the concept of Excessive Agency to describe situations where an LLM-based system can perform harmful actions as a result of unexpected, ambiguous, or manipulated outputs. The organization identifies three main causes: excessive functionality, excessive permissions, and excessive autonomy.

The difference between these dimensions is important.

An agent presents excessive functionality when it has tools or functions it does not need to perform its job.

There is excessive permissions when the necessary tools operate with privileges higher than required.

Excessive autonomy appears when the agent can execute high-impact operations without additional controls or approval.

These conditions can combine. An agent manipulated through prompt injection represents a greater problem if it also has an unnecessary tool, credentials with broad permissions, and autonomy to use it without confirmation.

The risk surface can be analyzed through the different layers involved in the system:

The matrix shows an important characteristic of AI Security: no isolated control covers the entire surface. Protection needs to combine controls over instructions, data, identity, tools, actions, and behavior during execution.

How to protect agents and applications with an AI security strategy

A defensive architecture must assume that a model may encounter ambiguous or malicious instructions and that some preventive mechanism may fail.

A defensive architecture should limit both exposure to problematic content and the consequences it may produce if that content influences an agent's decision.

This favors a layered defense strategy where identity, authorization, guardrails, human oversight, and observability reduce different parts of the risk.

Identity, permissions, and isolation of agents

Each agent should have an attributable identity and a defined set of responsibilities.

Microsoft recommends using specific identities for agents, documenting what data and tools they can use, and reviewing their permissions when workflows, integrations, or deployment environments change.

This separation improves both prevention and investigation.

If multiple agents use shared credentials, it becomes more difficult to determine which component executed an action. A unique identity allows associating activity, permissions, and resources with a specific agent.

Authorization should also be applied at the moment of using a tool. Just because an agent has access to an integration does not mean all available operations should be authorized.

Microsoft recommends treating tool invocations as independent trust decisions and applying stricter controls when an operation is irreversible, has a high impact, or crosses a security boundary.

Guardrails and validation of inputs and outputs

Guardrails allow for setting additional limits around the behavior of an AI application.

They can detect potentially problematic inputs, restrict topics outside the system's scope, filter sensitive information, validate outputs, or introduce checks before using certain tools.

Their function is important, but they should not become the only defensive mechanism.

OpenAI recommends combining guardrails with authentication, authorization, strict access controls, and conventional software security measures. It also suggests using multiple specialized defenses rather than relying on a single filter.

This is especially relevant against prompt injection. Detecting a malicious instruction is useful, but limiting the agent's permissions and tools reduces the consequences even when detection does not work as expected.

Human oversight for high-risk actions

Human oversight does not have to be introduced at every step of a workflow.

Its value increases when placed at points where a decision can generate significant consequences.

Financial operations, deletion of information, changes in permissions, sensitive communications, or difficult-to-reverse actions may require confirmation before execution.

OpenAI identifies sensitive, irreversible, or high-impact actions as one of the main scenarios in which it is advisable to transfer control to a person.

Microsoft applies a similar criterion by recommending specific authorization and new approvals for irreversible or high-impact actions.

The result is graduated autonomy. The agent can automatically resolve lower-risk operations and request intervention when the potential impact exceeds the limits defined by the organization.

Logging and observability of agent actions

Logging only the conversation with the model provides an incomplete view of what occurred.

An investigation needs to reconstruct which identity acted, what permissions it had, what tool it used, what operation it requested, and on what resource it was executed.

Microsoft recommends logging elements such as agent identity, role, effective scope, action, resource, and correlation data that allow tracking the workflow from end to end.

This traceability becomes more important in architectures where an agent uses multiple tools or where multiple agents participate in the same process.

Observability must allow moving from a final response to the sequence that produced it.

AI security in production: what a company should monitor

The controls defined during development can lose effectiveness if the system changes after deployment.

An agent may incorporate new tools, access additional data sources, or take on responsibilities that did not exist in its initial version. Each change can modify its risk surface.

Monitoring should observe both obvious incidents and changes in the system's capabilities.

An unexpected increase in calls to a tool, repeated attempts to access resources outside the scope, frequent escalations to privileged actions, or unusual execution sequences may indicate problems that do not appear when analyzing only the model's response.

There should also be an updated inventory of agents and responsible parties. Microsoft recommends logging agents to know what systems exist, who is responsible for them, and what resources they can access. Unmanaged agents fall outside normal review, monitoring, and revocation processes.

The incident response should also include mechanisms to quickly reduce an agent's capabilities. Revoking credentials, invalidating tokens, disabling identities, or withdrawing permissions may be necessary when anomalous behavior is detected.

AI Security in production therefore requires maintaining a continuous relationship between observed behavior and effective permissions.

What profiles are involved in the security of AI systems

The protection of AI systems connects technical decisions with risk policies and operational security processes.

An AI Governance Specialist may participate in defining policies, responsibilities, controls, and approval criteria applicable to Artificial Intelligence systems. Their work helps establish what levels of autonomy are acceptable, what actions require oversight, and how risks associated with each application are documented.

Governance should also accompany the evolution of the system. Incorporating a new data source, an additional tool, or greater permissions can change the risk profile even if the model used remains the same.

The SOC Analyst intervenes from the operational perspective. The activity generated by agents, identities, tools, and connected systems can be incorporated into monitoring, detection, and investigation processes to identify anomalous behaviors and respond to incidents.

This relationship connects two different levels of AI Security.

Governance establishes what should be allowed and under what conditions. Security operations help verify what is actually happening in production and respond when observed behavior deviates from those limits.

AI security must control what AI can do, not just what it can generate

The risk surface of an AI application changes when the model acquires the ability to interact with other systems.

An LLM can generate a response. A connected application can retrieve information. An agent can select tools, combine data, and execute actions.

Each level incorporates new security decisions.

Protecting these systems requires controlling who acts, what information it can use, what tools are available, what permissions it receives, and what operations it can execute. Higher-impact actions need additional limits, while observability must allow reconstructing the workflow when something does not work as intended.

AI Security thus evolves from protecting the model to protecting the entire system surrounding it. When Artificial Intelligence can act on business processes, security must also control the scope and consequences of those actions.