# The Agent Release Checklist

*One threshold question, then fifteen more, before an AI agent goes into regulated work.*

This is a plain-language checklist for the person whose name is on the sign-off — the VP of Claims, the Head of Legal Ops, the Clinical Informatics lead, the compliance owner. It turns the questions that decide whether an agent is safe to run into rows you can score in a spreadsheet. It is tool-neutral: you need a careful reader and an hour, nothing else. **Question 0 decides which rules apply; questions 1–15 decide whether it ships.** Use it, copy it, adapt it, hand it to your vendor and ask them to fill it in.

**How to score it.** Give every row two cells, not one: a **verdict** — *Yes / No / Needs work* — and an **evidence** cell that names the artifact behind the verdict (the table, the log, the test result, the number and the bar it cleared). For the measured or tested questions (8, 9, 11, 13) a "Yes" is not honest unless the evidence cell names the figure and the threshold it met. A blank evidence cell is a "Needs work." A blank scorecard companion (CSV — one row per question, with verdict, evidence, and owner columns) ships alongside this checklist if you'd rather not build the sheet by hand.

**Hard stops vs. needs-work.** Rows marked **⛔** are hard stops: a "No" means the agent does not ship. Rows marked **⚠** are "raise it and fix it before you'd want to defend it." The line is drawn where a failure is a safety or legal breach rather than a maturity gap. Severity can shift with context — a ⚠ can escalate to ⛔ inside a specific regime, and Question 5's ⛔ relaxes to ⚠ when grounding isn't required; those shifts are noted on the row.

**About the tags.** Each question carries a *maps to* tag pointing at the **NIST AI RMF**, the **OWASP Top 10 for LLM Applications (2025)**, and the **EU AI Act**. Those tags are pointers to the framework and to the artifact an auditor will ask for — they are not, by themselves, the evidence. The filled-in sheet tells you where you stand; the underlying logs, tables, and test results are what you hand to the auditor.

---

## 0. Classification: do you know which rules apply? ⛔

**Is the agent's regulatory status decided and written down — whether it is a "high-risk" system, and which laws govern each decision it touches?**

A good answer names the use-case category (insurance pricing or claims, creditworthiness, access to essential services, employment, healthcare, law enforcement…), states whether it is high-risk under the EU AI Act's Annex III, and — if you claim it is *not* high-risk — records the reason. Map each regulated decision the agent makes to the law that governs it. This question decides which of the rows below are mandatory for you.

**Red flag:** "We'll figure out compliance later." Classification is the first decision, not the last — it determines everything downstream.

*Maps to: EU AI Act Art. 6 & Annex III (classification), Art. 6(3) (non-high-risk rationale); NIST GOVERN-1.1 (applicable law mapped).*

## 1. Decide vs. recommend: is the line written down? ⛔

**For every action the agent takes, is it recorded whether it may *decide* on its own or may only *recommend* to a human?**

A good answer is a short table: this class of action the agent completes; that class it only proposes, and a named human commits it. The most consequential and irreversible actions — pay a claim, deny coverage, sign a contract, change a patient record — belong on the *recommend* side unless you can defend automating them.

**Red flag:** "It's smart, we let it handle the routine ones" — with no written line between what it decides and what it merely suggests.

*Maps to: EU AI Act Art. 14 (human oversight); NIST MAP-2.1/2.2 (autonomy scoping); OWASP LLM06 (excessive agency).*

## 2. The "must not" list: are the hard boundaries written as nevers? ⛔

**Does the agent carry an explicit, written list of things it must never do — and is that list non-empty?**

A good answer names concrete prohibitions in the agent's own terms: never approve payouts, never deny or delay a claim, never contact a claimant, never alter policy or patient data. These are the hard limits it is forbidden from crossing, and the basis for an automatic refusal when something tries to push it across.

**Red flag:** An empty list, or vague aspirations like "act ethically" and "use good judgment" — nothing a machine or an auditor can point to.

*Maps to: NIST MAP-1.1 (prohibited/out-of-scope uses); EU AI Act Art. 13 (limitations documented); OWASP LLM06.*

## 3. Rule of Two: is at least one risky power switched off? ⛔

**Is at least one of these three powers deliberately switched off — reading private data, taking in untrusted or outside content, and sending data out? (Holding all three at once is a hard stop.)**

An agent may safely hold at most two of the three. A good answer shows one leg turned off on purpose: it reads the private file and drafts internally but cannot send outbound; or it reads outside content and sends, but touches no private data. The combination of all three is the *lethal trifecta* — all three data-powers on at once, the pattern an attacker exploits to turn a poisoned document or email into a data leak. Cap it at two, and a prompt-injection attack has nowhere to send what it steals (to *exfiltrate* = quietly ship your private data out).

**Red flag:** "It reads the inbox, browses the web, *and* sends email" — all three legs on.

*Maps to: OWASP LLM01 (prompt injection) & LLM02 (sensitive-information disclosure); NIST MEASURE-2.7 (adversarial/injection testing); EU AI Act Art. 15 (resilience to manipulation).*

## 4. Mandatory budgets: are there hard, tested caps per run? ⛔

**Does a single run have enforced ceilings — a maximum token count, a maximum number of steps, and a maximum cost — plus, if the agent can spawn sub-agents, a cap on how many and how deep?**

A good answer is real numbers, present and non-negotiable, and *tested* — a run that trips a ceiling should be observed to stop. This is the circuit breaker against a runaway loop or a *denial-of-wallet* event (an unattended loop that runs up a huge bill).

**Red flag:** "It usually costs about a penny" — an estimate instead of an enforced cap, one of the ceilings missing, or an uncapped sub-agent fan-out.

*Maps to: OWASP LLM10 (unbounded consumption); NIST MANAGE-2.3/2.4 (disengage/deactivate); EU AI Act Art. 15 (robustness).*

## 5. Grounding, citations, and review of consequential claims ⛔ *(when grounding is required; ⚠ otherwise)*

**When the agent must be grounded in real sources, does a check verify every answer cites retrievable evidence — and do consequential factual outputs get a human read even when grounded?**

A good answer is a pair: a rule that answers must be backed by retrieved sources, *and* an output check that fails a response with a missing or invented citation. A grounded-but-wrong answer — a real source misread — still needs a human on consequential claims; name automation bias as the thing you are guarding against. (If grounding is *not* marked required, an absent citation check is a needs-work, not a hard stop.)

**Red flag:** "We told it to use our knowledge base" — grounding asserted, nothing verifies it, so it can still cite a document it never read.

*Maps to: OWASP LLM09 (misinformation/hallucination); NIST MEASURE-2.3 (validity vs. ground truth); EU AI Act Art. 15 (accuracy) & Art. 14 (automation bias).*

## 6. Least privilege: which tools can it call, and which need a human OK? ⚠ *(⛔ for irreversible regulated actions)*

**Is every tool the agent can use listed, and does every tool that *changes* something require a human approval?**

A good answer is a short inventory: read-only tools it may use freely, and every write tool — send, alter, move money, commit a record — flagged as requiring a person to approve the action. An unguarded write tool is a warning to fix in general — but for an irreversible regulated action (pay a claim, sign a contract, alter a record, move money) a missing approval is a hard stop.

**Red flag:** A write tool that fires on its own — "it can send the email itself" — with no approval gate in front of a real, often irreversible action.

*Maps to: OWASP LLM06 (excessive agency) & LLM05 (improper output handling); NIST MANAGE-1.3; EU AI Act Art. 14.*

## 7. Evidence, audit trail, and redaction ⚠ *(⛔ in records-heavy regimes)*

**Is every consequential action logged so a reviewer can reconstruct it — with sensitive data redacted, a stated retention period, tamper-evident storage, and each record attributed to the acting agent or human?**

A good answer declares concrete rules: every decision cites the rule it relied on; inputs, retrieved sources, tool calls, and outputs go to a durable, reviewable record (a log an auditor can retrieve later). Two cautions: the log must be **redacted and minimized** — do not write PII/PHI verbatim, or the audit trail becomes the leak — and it must be **retained and tamper-evident** for the regime's required period. In records-heavy regimes (pharma electronic records, audit work papers) a missing or unretained trail is a hard stop.

**Red flag:** "We can check the chat history" — no declared rules, no retention, no redaction, nothing you could hand a regulator to reconstruct one decision.

*Maps to: EU AI Act Art. 12 (logging) & Art. 26 (deployer logs); NIST MANAGE-4.1/4.3; OWASP LLM02 (data disclosure via logs).*

## 8. Refusal correctness: does it reliably decline what it should? ⛔ *(measured)*

**Is there measured proof the agent refuses out-of-scope, prohibited, and adverse-action requests instead of overreaching?**

A good answer is a tested number: the agent correctly declines the cases it must decline (anything requiring a licensed human, anything on the "must not" list) at a high, written bar. For regulated claim or credit decisions, an *adverse action* (deny, reduce, delay) routes to an accountable human every time — and that boundary is backed by a real control, not just prose.

**Red flag:** "It knows not to do that" — a belief, not a measurement, with no refusal rate tested against real out-of-scope cases.

*Maps to: NIST MEASURE-2.5 (reliable in deployment condition); EU AI Act Art. 14; OWASP LLM06.*

## 9. Prompt-injection resistance: can untrusted input make it escalate? ⛔ *(tested)*

**Has the agent been deliberately attacked with hostile input — a poisoned document, email, or web page that tries to override its instructions, extract its system prompt, or exfiltrate data — and held its ground?**

A good answer describes red-team testing of both direct instructions and indirect ones hidden inside content the agent reads, with the agent holding its boundaries; model output is treated as untrusted before it reaches any downstream system. Confirm the system prompt and guardrails resist extraction, and that no secrets or authorization logic live in the prompt.

**Red flag:** "No one would put instructions in a PDF" — no adversarial testing, and untrusted content flowing straight into privileged actions.

*Maps to: OWASP LLM01 (prompt injection), LLM05 (output handling) & LLM07 (system-prompt leakage); NIST MEASURE-2.7 (red-team); EU AI Act Art. 15 (cybersecurity).*

## 10. Human oversight: can a person review, override, and stop it? ⛔

**Can a named human understand the output, intervene, override a decision, and stop the agent — and has the stop path actually been exercised?**

A good answer shows a working, *tested* stop button and takeover path: a person can pause the workflow, reverse or block an action, and the interface guards against blindly trusting the machine on regulated decisions. "Working" means a run was halted and observed to stop, not that a button exists in principle.

**Red flag:** "By the time anyone sees it, it's already done" — no intervention point before a consequential action commits.

*Maps to: EU AI Act Art. 14 (human oversight) & Art. 26 (deployer duties); NIST MANAGE-2.3/2.4 (kill switch), MEASURE-2.13 (oversight effectiveness).*

## 11. Regression proof: must a saved test suite pass before each release? ⛔ *(measured)*

**Is there a saved set of test cases — including known red-flag and adverse cases — that the agent must pass before *every* release, not just the first?**

A good answer is a stored suite with written thresholds. In plain terms: *recall* = of the true red-flag cases in your test set, the share the agent caught; catching them all means it missed none **in the suite**. For safety-critical work (patient triage — signs of stroke, sepsis) hold red-flag recall at 100% of the suite, where one missed case turns it red — and keep expanding the suite, because a finite test set bounds known failure modes, not all of them. Nothing ships without a green run.

**Red flag:** "It passed when we built it" — no re-run before each release, so a quiet model or prompt change can regress unnoticed.

*Maps to: NIST MEASURE-1.1/2.5 (metrics vs. ground truth); EU AI Act Art. 9 (lifecycle risk management) & Art. 15 (accuracy); OWASP LLM09.*

## 12. Data governance: is the corpus representative, sourced, and scoped? ⚠ *(⛔ for retrieval across tenants)*

**Is the data the agent is built on and retrieves from governed — representative, provenance-checked, and access-scoped?**

A good answer shows the training, tuning, and retrieval (RAG) corpus is examined for representativeness and bias against the affected population; that sources have checked provenance and integrity; and that retrieval is access-controlled per user or tenant with defenses against poisoning. An agent can cite a document it retrieved (question 5) while retrieving another tenant's records or a poisoned source — retrieval that crosses tenants is a hard stop.

**Red flag:** "It just searches all our documents" — no tenant scoping, no provenance, no bias examination.

*Maps to: EU AI Act Art. 10 (data governance); NIST MAP-2.3, MEASURE-2.11; OWASP LLM04 (data poisoning) & LLM08 (vector/embedding weaknesses).*

## 13. Fairness and non-discrimination: is it tested on people? ⛔ *(measured, for decisions about people)*

**For decisions about people — claims, credit, employment, benefits — is the agent tested for discriminatory outcomes across protected classes and their proxies?**

A good answer is a documented bias evaluation across protected classes and known proxies (zip code, surname, name), a stated disparity threshold, and a control that flags an adverse decision that turns on a protected class or a proxy. Accuracy on average is not fairness across groups.

**Red flag:** "We don't use protected attributes" — proxies leak them, and untested is unproven.

*Maps to: NIST MEASURE-2.9/2.11 (fairness/bias); EU AI Act Art. 10 (bias examination) & Art. 15.*

## 14. Post-market monitoring and incident response ⚠

**Once it's live, is there a plan to watch it, catch drift, report serious incidents, and roll it back?**

A good answer is a production-monitoring plan for accuracy and drift and for whether the human oversight is actually working; a serious-incident detection and reporting workflow; and a rollback/recovery plan. A pre-release gate is necessary, not sufficient — models and inputs move after launch.

**Red flag:** "We'll keep an eye on it" — no monitoring plan, no incident path, no rollback.

*Maps to: EU AI Act Art. 72 (post-market monitoring) & Art. 73 (incident reporting); NIST MANAGE-4.1/4.3 & 3.2, MEASURE-2.13.*

## 15. Accountable owner: does a named human sign the release? ⛔

**Is there one named, accountable person — not a team, not "engineering," not "the platform" — who signs this release and owns what the agent does next?**

A good answer names the individual, their authority to say no, and the escalation and stop path when the agent fails in production. Sign-off is a personal act of accountability, not a checkbox.

**Red flag:** "The platform approved it" — no human name on the line, and no one who can be held responsible for an incident.

*Maps to: NIST GOVERN-2.1/1.2 (accountable owner, named roles); EU AI Act Art. 16–17 (provider obligations) & Art. 26; MANAGE-4.3 (incident response).*

---

## The Human Decision Plan

The checklist above tells you whether the agent is *safe to run*. This section tells you *who is in charge of it*. Before sign-off, write down five things in plain language — one page is enough.

- **What the agent may decide on its own.** The low-stakes, reversible, well-tested actions where automation is defensible: routing, drafting, classifying, flagging for review. If an error here is cheap and quickly caught, the agent may complete it.

- **What the agent may only recommend.** The consequential and irreversible actions — approving or denying a claim, reducing or delaying a benefit, signing a contract, stating a diagnosis or recommending medication, altering a record, moving money. The agent may prepare and propose; a named, qualified, and where required *licensed* human commits the action. Adverse actions against a person (deny, reduce, delay) always live here.

- **Where a human must stay in the loop — every time, not on a sample.** The specific decision points where a person must see and approve before anything commits: any write tool firing, any adverse action, any red-flag or emergency case, anything the agent itself flags as low-confidence or out-of-scope.

- **What the people affected are told.** Where a claimant, patient, or customer is on the other end, they are told they are dealing with an AI, AI-generated content is labeled where required, and there is a channel to reach a human and contest the outcome.

- **Who decides, and who is accountable when it fails.** Name the individual who owns each tier above, the person who signs the release, and the escalation and stop path when the agent misbehaves in production. Authority to *pause the agent* should sit with someone who can act within minutes, not days.

A useful test: read this page to someone who has never seen the agent. If they cannot tell you who to call when it goes wrong, the plan is not finished.

---

## If your agent is high-risk (EU AI Act): the formalities

Only if Question 0 landed on *high-risk*, these formal obligations sit outside the fifteen questions and still have to be met before you place the system on the market:

- **Technical documentation and instructions for use** are complete and current — intended purpose, deployment context, capabilities, limitations, accuracy metrics, known failure modes (Art. 11 & Annex IV; Art. 13).
- **Fundamental Rights Impact Assessment** completed where required — public bodies and certain regulated services, before first use (Art. 27).
- **Conformity assessment, EU Declaration of Conformity, CE marking, and EU-database registration** done before going to market (Art. 43 / 47 / 48 / 49).
- **A Quality Management System** with assigned provider responsibilities is in place (Art. 16–17).
- **Third-party model and tool governance:** the vendor model and tools' licenses permit the regulated use, provenance/integrity is documented, and there is a plan for a vendor or model change (NIST GOVERN-6.1, MANAGE-3.1; OWASP LLM04).

---

## What this checklist does *not* cover — read this too

This checklist makes agents safer where they belong. It does not make an agent the right tool, and it does not catch everything.

Situations where a human, or plain observability, beats an agent:

- **The decision is rare, high-stakes, and irreversible.** A few times a week, and each instance can end a life, a livelihood, or a legal position — a contested claim denial, a diagnosis, a termination. Low volume plus high stakes rarely justifies the governance overhead an agent demands; a trained human doing the work, not an agent recommending and a human rubber-stamping, is the honest answer.

- **You cannot yet describe what a right answer looks like.** No ground-truth test set means questions 8, 11, and 13 are unanswerable — you cannot measure accuracy, refusal, red-flag recall, or fairness. Build the test set and let humans work the cases first. An unmeasurable agent is not ready, however good the demo looked.

- **The real problem is that no one can see what is happening.** Sometimes the need is not autonomy but *visibility* — a dashboard, logs, and alerts so a human catches problems sooner. Observability over an existing human process often delivers more safety, faster, than inserting an agent into the loop.

- **The workflow is unstable or the rules are in flux.** If the policy, regulation, or process changes month to month, a saved test suite and a fixed boundary list cannot keep up, and an agent quietly drifts out of compliance. Wait for the ground to settle.

And be honest about the instrument itself: this is a governance-and-safety gate. If your dominant risk is model bias, data retention, vendor lock-in, or procurement, a fairness audit, a privacy review, or a vendor-risk assessment will catch things this list does not. A "not yet" here is a real answer, not a failure of the checklist — and sometimes the checklist is simply the wrong instrument for your risk.

---

*Free to use, copy, and adapt (CC BY 4.0). Published by Assize — assize.ai. This is a governance aid, not legal advice; your counsel judges the evidence.*