TransparencyGovernance

We let AI audit our compliance reports. They failed.

Every compliance formatter we had was overclaiming. We found out the same way an auditor would: by checking the claims against the actual regulatory text.

10 April 20266 min read

An AI eye examining its own reflection critically, amber-gold tones on black, circuit board textures dissolving

A HIPAA score of 100 with zero audit events recorded. An ISO 27001 assessment covering 16 of 93 controls. An EU AI Act report with wrong article numbers. These were our own compliance reports. This is what we did about it.

Enterprise software companies that build compliance tooling face a particular problem: the people building the formatters are engineers, not compliance lawyers. The natural instinct is to map what the system can observe to what the framework asks for. Sometimes that mapping is correct. Often it is not.

We had five compliance formatters: GDPR, HIPAA, SOC2, ISO 27001, and the EU AI Act. We believed they were reasonably accurate. Then we ran them against GPT-4 with the actual source regulations as context, using an explicitly adversarial prompt. The model's job was not to confirm what looked correct, it was to find what was wrong.

Every formatter had Critical or High findings. This is the story of what was wrong, why it happened, and what credible compliance reporting actually looks like.

Why we ran an adversarial audit at all

The trigger was a specific pattern we noticed in our GDPR formatter: it marked transparency as compliant based on internal event logging. That felt wrong. GDPR transparency requirements face outward: they are about informing data subjects, not maintaining internal records.

Once we saw that one, we wanted to know how many others were like it. Rather than do a manual review (which would be slow and subject to the same blind spots that produced the original errors), we took the approach of treating the model as an adversary whose job is to find holes.

The value of this approach is that it is fast and it is genuinely adversarial: the model is not trying to help you feel better about your compliance posture. It has no relationship with the people who built the formatters. It simply checks claims against source material.

The methodology

Export compliance reports from the formatter. Pull current state reports for each framework, GDPR, HIPAA, SOC2, ISO 27001, EU AI Act, from the formatter system in their full JSON output.
Provide the actual regulatory text as context. Upload the source documents: GDPR full text (EUR-Lex), HHS HIPAA Security Rule, ISO 27001:2022 (licensed copy), EU AI Act (official journal). The model works against source, not summaries.
Adversarial review prompt. The prompt explicitly instructs the model to find what is wrong, not to confirm what looks correct. "Identify every item where the formatter is asserting compliance or completion without sufficient evidence, and every place where the chosen status does not match regulatory requirements."
Categorise and triage findings. Each finding categorised as Critical (misleads in a way that could cause harm), High (materially incorrect), Medium (incorrect but bounded), or Low (framing issue). Critical and High require immediate rewrite.
Rewrite formatters, then re-run the audit. Revised formatters are submitted to the same adversarial prompt. Any new findings restart the loop. Done when a full adversarial pass produces no Critical or High findings.

What the audit found

Five frameworks, five formatters, findings across every one. Here is a summary of the primary finding in each, what the regulation actually requires, and what the rewrite produced.

Framework	Severity	Originally claimed	Finding	What the regulation requires	Rewrite
GDPR	critical	Compliant	Transparency obligation addressed by internal logging	Articles 13 and 14 require proactive disclosure to data subjects: an outward obligation, not an internal metric	Split into data-subject-facing requirements (REQUIRES_ACTION until notification mechanism is confirmed) and internal controls (evidenced)
HIPAA	critical	100 / 100	Full score with zero audit events recorded	The Security Rule requires demonstrated implementation, not absence of recorded failures. Zero events means zero evidence, not zero violations.	Score gated on minimum audit event count; zero-event state now produces REQUIRES_ACTION for all audit controls
ISO 27001	high	Assessment complete	Assessment section covered 16 of 93 Annex A controls	ISO 27001:2022 Annex A has 93 controls across 4 themes. Partial coverage is not assessment completion.	Coverage counter now visible in report header; uncovered controls emit explicit CONFIGURE markers rather than being omitted
EU AI Act	medium	High-risk obligations addressed	Article references were incorrect: two citations pointed to the wrong articles	Article 14 covers human oversight; Article 15 covers accuracy and robustness. Both were cited under a single item labelled incorrectly.	Each article now cross-referenced against the published regulation text; formatter tested against known-correct citations
SOC2 Type II	high	Controls evidenced	Evidence mapped to availability of infrastructure, not to demonstrable trust service criteria implementation	SOC2 Type II requires evidence of control operation over time: a period of operation, not a point-in-time snapshot	Time-range requirement explicit in report; point-in-time assessments now labelled as readiness snapshots, not Type II evidence

What changed in the output

The two most significant rewrites were GDPR transparency and HIPAA audit controls. Both show the same pattern: a false positive that would survive casual review but fail under scrutiny.

GDPR transparency (before)json

{
  "control": "transparency",
  "status": "COMPLIANT",
  "evidence": "Chat processing logged in VOLT chain"
}

Conflates internal logging with data-subject notification obligation

GDPR transparency (after)json

{
  "control": "transparency",
  "status": "REQUIRES_ACTION",
  "evidence": "Internal processing: evidenced via VOLT chain",
  "gap": "Data subject notification mechanism not confirmed",
  "action": "Verify Articles 13/14 notices are delivered at data collection points"
}

Internal evidence cited; gap explicitly named; action item specific

HIPAA audit controls (before)json

{
  "control": "audit_controls",
  "status": "COMPLIANT",
  "score": 100,
  "audit_events_recorded": 0
}

100% score with no supporting evidence, the worst possible combination

HIPAA audit controls (after)json

{
  "control": "audit_controls",
  "status": "REQUIRES_ACTION",
  "note": "Score gated on minimum audit event count",
  "events_recorded": 0,
  "minimum_required": 1,
  "action": "Confirm audit logging is active and events are being captured"
}

Zero events produces action item rather than false confidence

What this means for enterprise buyers

If you are evaluating an AI governance platform, the right question to ask is not "do you have a compliance report?" It is: "have you had an adversarial review of that report's claims, and can I see the findings?"

A vendor who cannot answer that question has likely not done the work. A vendor who says the first version passed without any findings has either not tried hard enough or is using a narrow enough definition of the framework that everything passes by design. The findings themselves should be checked by an auditor, not just claimed.

We are publishing what we found because we think the industry needs a higher baseline for what "compliance reporting" means. Internal readiness assessment with honest scope statements, three-state output (evidenced, requires action, requires configuration), and at least one adversarial review pass. That is the minimum a serious platform should offer.

Publishing your failure modes sounds counterintuitive. But enterprise buyers who care about compliance are sophisticated enough to recognise that a vendor who found and fixed their own problems before deployment is more trustworthy than one who claims no problems were ever present. The audit findings are now part of our release notes. They will continue to be.
On transparency as a competitive advantage

Built for regulated industries

Compliance Suite ships with VOLT audit trails, WARD receipts, and AOCL policy enforcement. Self-hosted, air-gapped, yours.

Compliance Suite Talk to sales