When an AI agent workflow fails, “the model made a mistake” is not a useful conclusion. An operations team needs to know what entered the workflow, which context and policy were active, what action actually occurred, what was prevented, and which control will make the next occurrence less likely.
This AI agent incident postmortem template is for a completed incident or near miss in a business workflow. It is not an outage runbook: use the runbook to contain and recover first. Use this record after the situation is stable to turn traceable facts into bounded corrective actions and regression checks.
What makes an agent incident different
A conventional system review can focus on a service, deployment, or dependency. An agent workflow also has an interpretation layer: instructions, retrieved context, model and version, tool permissions, approval state, and the decision that selected an action. Capture those as evidence, not recollection. A useful AI agent audit trail makes that reconstruction possible.
Keep the boundary clear. The agent interprets inputs and proposes or selects within its permitted scope. Deterministic rules validate required fields, limits, permissions, and state transitions. A human decides exceptions and high-impact actions. Systems persist the record and execute authorized side effects.
Start with containment, then preserve evidence
Do not begin a review by changing prompts or retrying the workflow. First stop further harmful side effects, record the incident boundary, and preserve the evidence needed to explain it. AWS guidance recommends AI-specific observability and emergency shutdown capability for high-risk scenarios; its Well-Architected guidance also recommends a standardized, blame-free analysis process.
- Contain: disable the risky capability, move to a safe fallback, or require human approval. Record who made that decision and when.
- Preserve: retain the input, retrieval references, prompt or instruction version, trace and tool-call IDs, authorization decision, approval record, output, and external-system response.
- Reconcile: determine whether each attempted side effect succeeded, failed, or is unknown. Unknown is a state to investigate—not permission to repeat.
- Restore deliberately: re-enable only after the owner accepts a tested correction and a rollback path exists.
For repeatable actions such as ticket creation or payment changes, record the idempotency key, state before the attempt, and external reference. The idempotency design should prevent a recovery attempt from duplicating an action whose outcome is uncertain.
AI agent incident postmortem template
Complete the template from logs, traces, tickets, and system records. Mark an unavailable fact as unknown; do not fill gaps with a plausible narrative.
| Section | Record | Completion condition |
|---|---|---|
| 1. Incident boundary | Incident ID, severity, affected workflow, owner, start and end times, customer or operational impact. | The business impact and review owner are explicit. |
| 2. Intended work | Trigger, expected output, approved scope, policy or business rule in effect, and whether human approval was required. | A reviewer can tell what the agent was allowed to do. |
| 3. Evidence timeline | Timestamped trigger, context retrieval, agent decision, tool calls, approvals, external responses, detection, containment, and recovery. | Each material event points to a trace, log, ticket, or system record. |
| 4. Active configuration | Instruction or prompt version, model and version, retrieval source/version, tool principal, permission scope, workflow version, and rollout state. | Active artifacts can be reproduced or identified as unavailable. |
| 5. Side-effect ledger | For each external action: target, authorization, idempotency key, result, and reconciliation state: succeeded, failed, or unknown. | No ambiguous action is silently replayed. |
| 6. Control analysis | Immediate cause, contributing factors, and whether the gap was context, policy, validation, approval, permission, or monitoring. | The explanation identifies a mechanism, not a person or model alone. |
| 7. Corrective actions | Action, owner, due date, affected artifact, verification method, and rollback condition. | Every action is testable and owned. |
| 8. Learning loop | Regression case, eval update, runbook or policy update, rollout gate, and closure evidence. | The incident changes the system that allowed it. |
Worked example: duplicate supplier outreach
Illustrative example: an intake agent reads a vendor request and calls an outreach tool twice after a timeout. The second call returns a different request ID, so the team initially assumes both messages were delivered.
The review should establish whether the tool accepted an idempotency key, whether the workflow persisted the first request ID before the timeout, whether the authorization policy allowed a second send without review, and whether monitoring flagged duplicate targets. The corrective record might require a deterministic duplicate-target check before send; persisted side-effect state; a reconciliation queue for unknown delivery state; a regression case that simulates timeout-after-acceptance; and a rollout gate that proves the case passes before re-enablement.
Turn findings into a closed control loop
A postmortem is complete only when its findings influence operating behavior. AWS’s Agentic AI Lens specifically warns that reviews which are simply filed fail to compound learning. For each finding, choose one destination:
- Policy: tighten a permission, approval threshold, or scope rule.
- Workflow: add deterministic validation, a required persisted state, a timeout branch, or safe fallback.
- Evaluation: add the incident input and expected safe behavior to a regression suite.
- Observability: create a signal that detects the failure earlier and names the owner.
- Runbook: update containment, reconciliation, or re-entry steps when they proved unclear.
Measure the loop with evidence: percentage of material events linked to a trace, time from detection to containment, unknown side effects reconciled before replay, corrective actions verified by their due date, and regression cases that pass before rollout. These are operational measures, not promised savings.
Where Workhint fits
Workhint can help an operations team design a connected work system around an AI-assisted process: route intake, retain workflow state, assign role-based approvals, manage exceptions, capture evidence, and coordinate follow-up work. The AI model remains a bounded interpretation component; Workhint is the orchestration layer for the surrounding workflow, records, permissions, and human decisions. Book a demo to map the controls and recovery path around a real operational workflow.
Before you close the review
- Can every consequential claim be traced to a system record or marked unknown?
- Did the review distinguish agent interpretation from deterministic rule enforcement and human judgment?
- Are uncertain side effects reconciled rather than replayed?
- Does every corrective action have an owner, due date, verification, and rollback condition?
- Has the incident become a regression case before the changed workflow is broadly re-enabled?
A concise, evidence-led review is more useful than a long narrative. Its job is to make the next similar workflow safer to operate.
Leave a Reply