Console/Knowledge/AI agent security
21 / 27

AI agent security testing across prompts, tools, memory, and actions

AI agent security testing evaluates the AI system that makes decisions and takes actions, not the security team’s use of AI to run a pentest. The scope includes prompts, tool calls, memory, identity, data access, sandboxing, approvals, and the evidence that connects an unsafe decision to a real side effect.

Direct answer

Test an AI agent with adversarial inputs, controlled tools, synthetic data, and explicit stop conditions. Require a trace from the triggering input through model behavior, tool execution, authorization, and observed impact before promoting an issue to a confirmed vulnerability.

WORKSPACE / EVIDENCEreviewable
Isometric AI agent security test showing prompts, reasoning, tools, memory, identity, data boundaries, and evidence
Authorized targets only · human approval available
Primary riskPrompt injection and excessive agency
Required scopeAgent, tools, memory, data, and runtime
Acceptance ruleReproduced action plus trace evidence
Definition

What is AI agent security testing?

AI agent security testing is an authorized assessment of how an autonomous or semi-autonomous AI system behaves when inputs, tools, memory, identities, or external data attempt to cross a security boundary. It examines the whole agent runtime rather than only the language model response.

An agent combines a model with instructions, context, tools, credentials, state, and an execution environment. The model may select a function, browse a resource, call an API, alter memory, or delegate to another agent. Each transition is a security decision that needs a testable policy and an observable record.

The useful result is not a collection of unusual prompts. It is a reproduced chain showing that untrusted input changed the agent’s objective, caused an unauthorized tool call, exposed protected context, or produced an action outside the approved policy. Reviewers need enough evidence to distinguish prompt sensitivity from a real security impact.

ARTEX treats agent security as an evidence-led engineering problem. Define identities, tool permissions, data sources, stop conditions, and approval gates before testing. Then connect each probe to model input, runtime decisions, tool arguments, authorization checks, and the resulting state change.

Boundary

How is this different from AI-powered penetration testing?

AI-powered penetration testing uses agents defensively to test an application, network, cloud account, or API. AI agent security testing reverses the subject: the AI agent itself becomes part of the attack surface and must be tested for insecure behavior.

The distinction matters for scope and ownership. A pentest agent testing a web application needs bounded target access. An AI agent security assessment needs test prompts, tool manifests, memory and retrieval systems, model configuration, identity boundaries, runtime controls, logs, and safe synthetic data.

Some controls overlap. In both cases, authorization must be enforced outside the model, tools need least privilege, sensitive actions need approval, and evidence must remain available after the run. The difference is that agent security also evaluates malicious or contaminated context that may influence the agent’s own decisions.

Do not merge the two pages or keyword sets. Searches for AI penetration testing usually ask how AI can perform security work. Searches for AI agent security testing ask how to protect and verify agents that plan, remember, call tools, and act.

Coverage

What attack surfaces should AI agent security tests cover?

Start with every place untrusted information enters the agent: user prompts, retrieved documents, tool descriptions, tool outputs, web pages, email, uploaded files, memory, other agents, and system-generated summaries. A direct user message is only one injection path.

Then map what the agent can influence or reach after context is contaminated. High-value boundaries include tool selection, arguments passed into tools, credentials available to a tool, tenant and user identity, data returned to the model, outbound network access, persistent memory, and the approval path for state-changing actions.

Test the agent’s authority as well as its behavior. A prompt injection that only changes a status message is lower risk than one that suppresses an approval, calls a raw command tool, or reuses a privileged token for another resource. The security question is what the agent can cause, not whether the text looks adversarial.

Cover multi-agent delegation and sub-agent boundaries when they exist. A trusted planner may pass contaminated context to a worker, or a worker may return an instruction that the planner treats as trusted. Preserve the delegation path and the policy applied at each step.

AI agent security test matrix
SurfaceTest questionEvidence required
Prompt and contextCan untrusted content change goals, policies, or output handling?Input, assembled context, response, and policy decision
Tool selectionCan the agent invoke a prohibited or unintended tool?Tool registry, model decision, arguments, and result
Tool executionCan arguments cross identity, tenant, path, or network boundaries?Authorization decision, request trace, and side effect
Memory and retrievalCan poisoned state influence a later task or another user?Write source, stored record, retrieval event, and impact
DelegationCan one agent mislead another or bypass a policy boundary?Delegation chain, inherited identity, and tool outcome
RuntimeCan the agent escape its sandbox or retain excessive access?Process, network, secret, and approval-control evidence
Runtime

How should teams test tools, identity, and agent actions?

Give each tool a narrow contract covering its caller identity, input schema, authorization policy, side effects, timeout, and audit requirements. Test whether the runtime enforces that contract when the model sends malformed arguments, attempts a different resource, or follows an instruction embedded in retrieved content.

Use a controlled environment with synthetic users, tenants, records, and external services. Replace production secrets with test credentials that have the same permission shape but no access to real customer data. Create at least one action that should be blocked and prove that the block occurs even when the prompt argues for an exception.

Test the identity chain from user to agent to tool. A confused-deputy flaw appears when the agent uses its own broad credential on behalf of a less privileged user. A token-passthrough flaw appears when a token intended for one audience or resource is accepted by another. Both require evidence from the authorization layer, not only the text response.

Exercise memory and retrieval as persistent state. Write a poisoned record through one workflow, start a separate task, and determine whether the record changes a decision or leaks across a boundary. Test deletion, expiry, tenant isolation, and the distinction between user-controlled memory and trusted system instructions.

Treat tool output as untrusted input. A backend response, web page, document, or other agent can contain instructions that the model follows. The runtime should label provenance, limit what output can influence policy, and keep high-impact actions behind deterministic authorization rather than model interpretation.

  • Scoped test identities for user, agent, tool, and service boundaries.
  • Allow-listed tools with a deliberate prohibited action available for safety tests.
  • Synthetic memory records that can be deleted after the run.
  • Mock external services for file, email, payment, and network side effects.
  • A kill switch and explicit stop conditions for loops or unexpected egress.
Guardrails

Which guardrails should an AI agent security test validate?

Validate input filtering, instruction hierarchy, tool authorization, least privilege, network isolation, secret scoping, output handling, rate limits, loop detection, and human approval. A guardrail is security-relevant only when it is enforced outside the model and tested against an attempt to bypass it.

Keep sensitive policy in deterministic code. The model can propose a tool call, but the runtime should decide whether the caller, resource, action, and data are allowed. Approval should show the exact action and expected effect, not a vague summary that the model can manipulate.

Test graceful failure. If the model is unavailable, a tool times out, context is truncated, or an approval expires, the agent should fail closed for sensitive actions. Preserve the original request and the reason the run stopped so a reviewer can distinguish a safe denial from an unexplained outage.

Monitor the quality of the guardrail as well as its existence. Measure blocked prohibited actions, approval latency, false blocks, repeated bypass attempts, unexpected outbound connections, and actions that required human correction. A policy without operational metrics will drift as tools and prompts change.

  1. 01
    Establish policy

    Define which identities, tools, resources, and actions are allowed for each agent role.

  2. 02
    Isolate execution

    Constrain the sandbox, network, secrets, filesystem, and external tool reach.

  3. 03
    Interrupt impact

    Require deterministic approval before state-changing or sensitive actions.

  4. 04
    Probe the boundary

    Use benign prompts and synthetic targets to prove that prohibited actions fail.

  5. 05
    Review the trace

    Connect the input, model decision, tool arguments, policy result, and side effect.

  6. 06
    Measure and retest

    Track control failures and repeat the original path after every material change.

Evidence

What evidence should an AI agent security finding preserve?

Preserve the test identity, agent version, model and provider configuration, prompt or source content, assembled context, tool manifest, tool call arguments, authorization decision, raw tool result, network trace where relevant, memory write, approval record, and observed impact.

Redact secrets and sensitive data while keeping the causal chain intact. A reviewer should be able to follow the finding from untrusted input to model behavior to enforced or missing control to a concrete resource or action. A screenshot of a surprising answer is not enough.

Capture model nondeterminism honestly. If the unsafe behavior appears intermittently, record the number of attempts, the inputs that reproduced it, model settings, and whether the effect depended on a particular tool response or memory state. Do not report a single lucky output as a deterministic vulnerability without qualification.

Store the evidence outside ephemeral model context. The original artifacts should remain available when the conversation is reset, the agent memory expires, or the provider model changes. Link the finding to the test case and the retest so the security decision survives future upgrades.

Summary

What makes AI agent security testing credible?

Credible agent security testing starts from a defined runtime and policy, uses controlled identities and tools, connects every probe to an artifact, and proves the resulting action or data exposure. It separates prompt sensitivity, control failure, and validated security impact.

Repeat the original chain after tool changes, model upgrades, prompt revisions, memory updates, permission changes, and new external integrations. Agent behavior can change without application code changing, so the regression surface includes prompts, model versions, retrieval content, and tool metadata.

Treat third-party context and tool output as hostile by default. The most durable architecture does not ask the model to be the final authorization boundary. It uses the model for reasoning while deterministic services enforce identity, resource scope, data access, and approval.

ARTEX can retain agent activity, tool use, traffic, findings, and approvals in one reviewable task record. Use that evidence to test the agent’s own controls before granting it access to production systems or high-impact tools.

FAQ

Frequently asked questions

What is AI agent security testing?

It is the authorized testing of an AI agent’s prompts, tools, memory, identity, data access, runtime, and approval controls to determine whether the system can be manipulated into unsafe or unauthorized behavior.

How is it different from using AI agents for pentesting?

AI-powered pentesting uses agents to test another system. AI agent security testing makes the agent itself the target and evaluates the decisions, tools, credentials, memory, and actions that surround the model.

What is the most important AI agent security control?

No single control is sufficient. Least-privilege tool authorization, external policy enforcement, isolated execution, human approval for sensitive actions, and traceable evidence must work together.

Can AI agent testing run in production?

Only with explicit authorization, scoped test identities, synthetic data, rate limits, safe tool mocks, and approval for state changes. Start in an isolated environment and expand after the controls are proven.

How should AI agent security findings be retested?

Repeat the same prompt or context path, agent version, memory state, identity, tool configuration, and action after remediation, then compare the new trace and authorization result with the original evidence.

Method

References and review notes

Continue
Inspect the workflow

Start with source. Test with authorization.

Review the public ARTEX repository, run it in an owned lab, and keep human approval on actions that can change a target.