AI security glossary

Plain definitions of the terms used across the tracks. Each entry names the track that covers it.

Definitions drafted with AI assistance. Reviewed and signed by [practitioner name] on [date].

Filter terms

40 of 40 terms

agent

An agent is software that uses a model to choose steps toward a goal, often by calling tools and reading their results. Its authority comes from the accounts, permissions and interfaces around it, not from the model alone. Defending an agent means controlling what it can reach, checking its actions and keeping evidence of what happened.

agentic AI

Agentic AI describes systems that use models to plan or select actions across multiple steps, rather than only return a single answer. The term does not establish how independent, reliable or powerful a system is. Security work starts with its actual tools, identities, data access and stopping conditions, not with a vendor’s use of the label.

AI bill of materials (AI BOM)

An AI bill of materials is an inventory of the components and dependencies used in an AI system. It may identify models, datasets, libraries, services, versions and their sources. It helps teams trace changes and investigate affected systems. An inventory alone does not prove that a model is trustworthy, that licenses permit a use or that every dependency is secure.

AI gateway

An AI gateway is an intermediary through which applications reach model services. It can centralize routing, authentication, usage limits and logging, depending on its implementation. It is also a security boundary that needs its own access controls and monitoring. Passing requests through a gateway does not establish that their contents are safe or authorized.

approval gate

An approval gate is a checkpoint that prevents an action or release until specified conditions have been met. Those conditions may include test results, a scope check or a human decision. A useful gate sits in the execution path and cannot be bypassed by the agent it constrains. Recording an approval after an action is not the same control.

confused deputy

A confused deputy is a system that has legitimate authority but is persuaded to use it for someone who lacks that authority. In an AI workflow, an agent might mistake instructions in retrieved content for an authorized request. The key defense is to check who may request an action and what data and permissions the action may use.

context window

A context window is the amount of input and working text a model can consider in a particular interaction, usually measured in tokens. It can include instructions, user messages, retrieved documents and tool results. Putting content in the window does not make it trustworthy. Truncation or summarization can also remove facts or constraints that a workflow depends on.

data poisoning

Data poisoning is deliberate contamination of data used to train, adapt or inform a system so that its later behavior changes. The affected material might be a training dataset or a retrieval collection. Defenders need to consider who can change those sources, how changes are reviewed and how suspicious behavior can be traced back to its inputs.

Covered in: AI Platform Security Engineer.

detection engineering for agents

Detection engineering for agents is the work of designing, testing and maintaining ways to recognize suspicious agent behavior. Useful signals can include identity use, tool requests, denied actions, data access and changes to a workflow. A detection needs representative tests and an investigation path. An unusual model response alone does not establish that an incident has occurred.

direct prompt injection

Direct prompt injection is an attempt to steer an AI application away from its intended instructions through input supplied directly to it, such as a user message. Whether the attempt matters depends on the surrounding system and the authority available to it. Defenses must constrain data access and actions, rather than assume a stronger instruction will always prevail.

evaluation (evals)

An evaluation is a structured check of how a system behaves against defined cases and criteria. Security evaluations can test authorization, information exposure, unsafe actions and recovery behavior. Results depend on the cases, environment and scoring method. Passing a test set is evidence about those tests, not proof that every future input or deployment will be safe.

excessive agency

Excessive agency is a condition in which an AI system can do more than the task safely requires. The excess can come from broad permissions, powerful tools or too little human oversight. A mistaken decision then has greater consequences. Reducing available actions, separating identities and requiring approval for sensitive changes can limit that exposure.

goal integrity

Goal integrity means keeping an agent’s work aligned with the authorized task when it encounters new messages, documents or tool results. Untrusted content should not quietly replace the user’s goal or expand its scope. Defenders examine how goals are recorded, how proposed actions are checked and how the workflow stops when instructions conflict or authority is unclear.

guardrail

A guardrail is a control intended to limit a system’s inputs, outputs or actions. It might be a permission check, validation rule, filter or approval requirement. The label does not tell you how strong the control is. Ask what it enforces, where it runs, how it fails and whether an agent or attacker can bypass it.

human verification

Human verification is a named person checking work against evidence rather than accepting an AI-generated result at face value. In sicurity.ai’s publication standard, a practitioner owns the outline and judgment, checks the material, runs the lab and signs the record before release. A stated verification requirement is not evidence that a particular draft has already been verified.

indirect prompt injection

Indirect prompt injection is an attempt to influence an AI system through material it encounters while doing another task, such as a document, website or tool response. The attacker does not need to be the authorized user. The security problem is treating that material as authority instead of data, especially when the system can access information or take actions.

jailbreak

A jailbreak is an attempt to make a model produce behavior its configured restrictions are intended to prevent. The term often concerns model responses, while prompt injection concerns instructions crossing an application’s trust boundaries; an attempt can involve both. Assessments should define the behavior being tested and its consequences instead of treating a surprising answer as a complete security finding.

MCP (Model Context Protocol)

MCP is a protocol for connecting AI applications to external capabilities and context, including tools and resources exposed by servers. A common interface does not make a server or its responses trustworthy. Security reviews still need to examine authentication, authorization, tool descriptions, execution boundaries and which information crosses between the application and each connected service.

memory integrity

Memory integrity means keeping stored information used by an agent accurate, attributable and resistant to unauthorized change. Memory may include saved preferences, summaries or facts reused across tasks. If untrusted material becomes a persistent instruction or false fact, later decisions can be affected. Controls should address who can write memory, how changes are reviewed and how errors are removed.

MITRE ATLAS

MITRE ATLAS is a knowledge base of adversary tactics and techniques relevant to AI-enabled systems, with supporting examples and related information. Teams can use it to organize threat discussions and assessment coverage. A technique’s presence in the knowledge base does not establish that a particular product is vulnerable or that a training exercise reproduces a documented incident.

model serialization attack

A model serialization attack exploits the way software saves, loads or interprets a model artifact. Some formats or loading paths can execute code or process unsafe content, so obtaining weights is not merely downloading passive data. Defenders examine artifact sources, loading behavior, integrity checks and isolation. A familiar filename or model name is not proof of a safe artifact.

model supply chain

The model supply chain includes the sources, people, services and software involved in producing and delivering a model and its dependencies. Risks can enter through artifacts, datasets, libraries, updates or hosted services. Defending it involves tracking origin and change, reviewing dependencies and deciding what may enter a deployment. A model’s published reputation does not replace those checks.

Covered in: AI Platform Security Engineer.

module

A module is a focused unit of sicurity.ai training built around a defined problem, learning objectives and practical work. The planning estimate is two to four hours per module, with additional practice time varying by learner. A catalog description is not access to a released lesson. Check the track’s module status and stated setup requirements before planning study.

NIST AI Risk Management Framework

The NIST AI Risk Management Framework is voluntary guidance for identifying and managing risks associated with AI systems. Its core functions are Govern, Map, Measure and Manage. It can help structure responsibilities and decisions across a system’s life cycle. Using the framework is not a NIST certification, and citing it does not demonstrate that particular controls are effective.

non-human identity

A non-human identity is an identity used by software or a workload rather than a person, such as a service account used by an agent. Its credentials and permissions determine what the software can access. Defenders need an owner, limited authority, credential handling and a way to revoke access. A model’s instructions are not a substitute for identity controls.

OWASP Top 10 for LLM Applications

The OWASP Top 10 for LLM Applications is a community resource describing major categories of security risk in applications that use large language models. It helps teams discuss threats and organize reviews, but it is not a complete threat model for every system. Names and ordering can change between editions, so identify the edition used when making a specific mapping.

practice exam

A practice exam is a learning assessment used to check understanding and identify gaps before further study or a separate certification attempt. sicurity.ai practice questions are independently authored, not actual GIAC questions. A practice score is not a certification result, a guarantee of passing or a validated prediction of an exam outcome unless supporting evidence has been published.

prompt leakage

Prompt leakage is the unintended disclosure of instructions or other prompt content to someone who should not receive it. The consequence depends on what was exposed and what the recipient can do with it. System prompts should not be treated as secure storage for secrets. Investigations should distinguish exposure of ordinary instructions from disclosure of credentials, private data or privileged context.

provenance record

A provenance record documents who created or checked material and how it was produced. For a sicurity.ai module, the intended record names the practitioner, disclosed AI contributions, model and version, verification work and sign-off date. A sample record illustrates the format only. It must not be described as a signed record for content that has not completed review.

red teaming (AI)

AI red teaming is an authorized effort to test how an AI-enabled system can fail or be misused and to turn findings into defensive improvements. The scope, data, targets and permitted actions must be agreed in advance. A useful result includes reproducible evidence and limitations. It is not permission to test other people’s systems or contact real targets.

retrieval-augmented generation (RAG)

Retrieval-augmented generation is a pattern in which an application retrieves material and supplies it to a model to help form an answer. The retrieved material can be useful without being authoritative. Security controls need to preserve the requesting user’s permissions, track the source of content and prevent documents from becoming instructions that authorize actions or disclose other users’ information.

scoped assessment

A scoped assessment is security testing bounded by explicit permission, named systems, allowed methods and stopping conditions. For the described sicurity.ai exercises, that means synthetic data and controlled, course-owned environments. Findings should record what was actually tested and what was excluded. An interesting result outside the agreed scope does not make the action that produced it authorized.

sicurity.ai credential

The sicurity.ai credential is a proposed independent assessment associated with the Agentic AI Defender track. It is not a GIAC certification, and its final rules, scoring, verification service and availability are not yet finalized. A planned credential or interface preview does not mean anyone has earned it. Consult the track’s current assessment note before relying on it.

SIFT

SIFT is the backend virtual environment intended to connect every sicurity.ai course lab to the AI resources its work needs. It provides the common foundation described across the tracks, not a claim of current lab access. Labs run in the learner’s own environment, and each module states its setup requirements and any costs before the learner starts.

system prompt

A system prompt is an instruction supplied by the application to shape a model’s behavior. It can describe the task, constraints or expected response style, but it is not an authorization boundary by itself. Applications must enforce permissions and validate actions outside the model. Sensitive credentials should not be placed in prompts on the assumption that instructions will keep them secret.

tool calling

Tool calling is a pattern in which a model proposes a request to a function or external service and the application decides how to execute it. The resulting data may return to the model for another step. The proposal is not authorization. The application must check the caller’s identity, allowed operation, arguments and consequences before executing a request.

tool gateway

A tool gateway is an intermediary that checks and routes requests to tools used by an AI system. Depending on its design, it can enforce identity, scope, argument rules and logging at a central point. It must not simply trust an agent’s assertion that a request is allowed. Its own permissions, availability and failure behavior also need review.

tool trust

Tool trust is the judgment about whether a connected tool, its advertised capabilities and its returned content can be relied on for a particular task. Trust should be limited to what has been checked, not granted because a tool is reachable. A tool response can contain untrusted instructions, incorrect results or sensitive data even when the connection is authenticated.

track

A track is a sequence of sicurity.ai modules organized around a particular part of the AI/ML Security Engineer’s work. The four tracks address AI platforms, agentic defense, controlled offensive assessment and security automation. They are connected learning paths, not four promises of current availability. Their release order, module access and certification limitations are stated on the individual track pages.

trust boundary

A trust boundary is a point where data or actions cross between components with different authority or assumptions. A retrieved document entering a model’s context and a model requesting a privileged tool action are examples. Marking the boundary helps identify where checks belong. Crossing it should not silently turn untrusted content into instructions, permissions or evidence of a user’s intent.