AccountabilityGovernance
What Happens When AI Agents Make Mistakes?
Building accountability into autonomous systems: because "the chatbot is a separate legal entity" is not a defence.
8 March 20266 min read
A chatbot invents a refund policy. Engineers leak source code to a model. An AI agent executes the wrong command in production. In every case, the same question follows: who is responsible, what exactly happened, and can you prove it?
The incidents that changed the conversation
These are not outlier events. They are early signals of a structural problem: the gap between what AI agents can do and what organisations can account for.
| Year | Entity | What happened | Outcome | Citation |
|---|---|---|---|---|
| 2024 | Air Canada | Customer service chatbot fabricated a bereavement discount policy that did not exist | Tribunal ruled airline liable: chatbot is not a "separate legal entity" | Moffatt v. Air Canada, CRT, February 2024 |
| 2023 | Samsung | Engineers pasted proprietary semiconductor designs and meeting notes into ChatGPT | Confidential data transmitted to external servers with zero audit trail | Reported March–April 2023 |
| 2026 | Industry-wide | 88% of organisations reported AI agent security incidents in the past year | The gap between capability and accountability continues to widen | Gravitee, State of AI Agent Security 2026 |
The accountability gap is structural, not hypothetical
The majority of organisations acknowledge their AI governance has not kept pace with AI adoption. The pattern is familiar to anyone who has studied corporate accountability: a powerful new capability emerges, organisations adopt it aggressively, controls lag behind, then something goes wrong at sufficient scale to force a reckoning.
We have seen this pattern before. Twice.
Two precedents that predict the future
SOX transformed financial controls from company policy into infrastructure. HIPAA made healthcare data access provable, not just logged. AI agents are now making decisions with comparable consequences, and the regulatory walls are closing in.
- 2002Sarbanes-Oxley Act
Financial audit trails become legally mandatory. CEOs personally certify accuracy.
- 1996HIPAA
Healthcare data access must be provable, not just logged, but attributable and auditable.
- 2024Air Canada ruling
Organisations held liable for AI agent actions. The chatbot is not a separate legal entity.
- 2026EU AI Act (high-risk)
Article 14: human oversight. Article 15: accuracy and robustness. Fines up to 7% global revenue.
Under common law, the doctrine of respondeat superior (the principle that employers are liable for the actions of their employees) almost certainly applies to AI agents operating under organisational authority. When an AI agent acts on behalf of an organisation, within parameters set by that organisation, using data provided by that organisation, the legal liability sits with the organisation.
A taxonomy of agent failures
Not all mistakes are the same. Understanding the failure categories is the first step toward building controls that actually work.
| Category | Description | Layer |
|---|---|---|
| Fabrication | Agent invents information, policies, or capabilities that do not existe.g. Air Canada chatbot promising non-existent refund policy | Output |
| Data leakage | Sensitive information transmitted to external systems without authorisatione.g. Samsung engineers sharing proprietary designs via ChatGPT | Input |
| Wrong execution | Agent runs the wrong command, tool, or action in a live environmente.g. Production config change without approval gate | Execution |
| Policy bypass | Agent circumvents governance rules through prompt injection or context gapse.g. Overriding safety controls via carefully crafted input | Governance |
| Silent failure | Agent fails without logging or alerting, the mistake is never detectede.g. Dropped audit entries, missing correlation IDs | Observability |
Why audit logs are not enough
Most enterprise AI platforms offer audit logging. This is necessary but insufficient. An audit log records that events occurred. It does not prove that the record has not been altered after the fact.
Consider the analogy from aviation. Commercial aircraft carry "black boxes" (flight data and cockpit voice recorders). When something goes wrong, investigators do not rely on the pilot's recollection. They reconstruct the entire sequence from tamper-resistant hardware that records continuously, independently, and immutably. AI agents need the equivalent.
Before (honour system)
- Incident response: "We checked our logs"
- Audit readiness: Statement about process
- Decision reconstruction: Grep through logs hoping to find related entries
- Tamper detection: Trust that nobody modified the database
After (cryptographic proof)
- Incident response: "Here is a cryptographically verified evidence bundle that any third party can independently validate"
- Audit readiness: Statement about proof
- Decision reconstruction: Follow the correlation ID and reconstruct the entire decision tree
- Tamper detection: Mathematically detectable if any event is modified, inserted, or deleted
What cryptographic accountability looks like
Building genuine accountability into AI agent systems requires three interlocking capabilities. These are not features to bolt onto an existing system. They are architectural requirements, designed in from the start, the same way SOX controls must be designed into financial systems.
- Immutable event chains (VOLT): every action recorded as a discrete event. SHA-256 hashing with previous-hash linking creates an append-only chain. Modify one event and every subsequent hash breaks.
- Layered observability (AOCL): 11 processing layers, from ingress normalisation through identity, routing, policy, planning, context, delegation, verification, to audit writeback. Each layer emits its own independently auditable trace.
- End-to-end correlation (AEE): every message carries a correlation ID linking it to the original request. Every response references its source. The result is a complete causal chain from input to output.
VOLT + AOCL + AEE → complete causal chain from input to output.
Deterministic replay: the next frontier
Cryptographic audit trails tell you what happened. The next capability (deterministic replay of tool interactions) tells you what would have happened under different conditions.
Capture a complete record of every input, model interaction, tool call, and policy decision. Then replay with one variable changed (a different policy threshold, a different model version, different permissions) and observe how the outcome changes. This is counterfactual analysis applied to AI operations, the same class of analysis that aviation investigators have used for decades.
{
"original_session": "SES_20260308_X1Y2",
"replay_mode": "tool_interactions",
"variable_changed": "policy_threshold",
"original_value": 0.7,
"counterfactual_value": 0.9,
"outcome_changed": true
}The cost of waiting
The companies that had strong internal controls before Sarbanes-Oxley passed faced modest compliance costs. The companies that had to retrofit controls into systems designed without them spent orders of magnitude more, and in many cases, the retrofit was never fully complete.
The same pattern will play out with AI agent accountability. Building proof into the system from the start is an engineering decision. Retrofitting it after a crisis is an existential one. That is why the record needs an independent witness (WARD), not just a system that vouches for itself.
Three questions every organisation deploying AI agents should be asking
- Can we reconstruct exactly what our AI agents did in any given session, with cryptographic proof that the record has not been altered?
- Can we trace a chain of decisions from initial input through every agent interaction to final output?
- If something goes wrong, can we replay the session and demonstrate what should have happened differently?
The gap between capability and accountability will close. The only variable is whether it closes by design or by enforcement.
Deploy QuoxCORE: free, self-hosted
AI agent orchestration with built-in governance. Docker Compose up and running in under five minutes.