AI Systems Have Multiple Surfaces
AI red teaming is not only prompt injection. Modern AI systems include prompts, retrieval pipelines, tools, plugins, memory, files, API connectors, browsing, code execution, and workflow automation. Each surface has different trust boundaries and different failure modes.
I start by mapping what the system can read, what it can write, what tools it can call, what data it can retrieve, and what decisions it is allowed to make. A model that only answers questions has a different risk profile from an agent that can modify tickets, run commands, send messages, or access internal documents.
Prompt Injection Is A Control Failure
Prompt injection matters because untrusted content can influence a model that has access to privileged context or tools. The risk is not the text itself. The risk is that the model may treat untrusted text as instructions and cross a boundary it should respect.
Useful tests distinguish between harmless instruction following and actual boundary crossing. Did the model reveal hidden context? Did it call a tool incorrectly? Did it modify data? Did it trust retrieved content over system policy? Did it execute an action based on attacker-controlled text?
Tool Boundaries Need Policy
Agents are strongest and riskiest when connected to tools. Tool calls should have clear schemas, permission checks, confirmation gates, redaction rules, and audit logs. A free-form shell tool is very different from a structured tool that only accepts a target, protocol, and read-only operation.
AI red teaming should test whether the agent can be pushed outside intended tool use. That includes argument injection, unsafe defaults, confused deputy behavior, data exfiltration through tool outputs, and hidden instructions in retrieved documents.
Retrieval Needs Source Awareness
Retrieval-augmented systems can mix trusted policy with untrusted content. A support article, uploaded file, website, issue comment, or document chunk may contain instructions that should not control the agent. The system needs to preserve source boundaries and decide what content is data versus what content is instruction.
Testing should include malicious documents, conflicting instructions, poisoned summaries, tool-call bait, and requests that require the model to cite or reason about source trust. The result should be evaluated by behavior, not only by the final answer text.
Evidence-Driven AI Testing
AI red team findings need evidence just like traditional findings. A strong report includes the system role, user prompt, retrieved content, tool configuration, observed output, boundary crossed, and why the behavior matters. If a tool was called, the report should include the tool name, arguments, and result.
The goal is not to collect strange model responses. The goal is to identify ways the system can misuse trust, leak data, perform unauthorized actions, or mislead users in a repeatable way. Repeatability and impact decide whether a behavior is a real finding.