When an AI audits an AI: defined parameters and where the human stays in the loop

The phrase "an AI checking an AI" makes some people laugh and other people nervous, and both reactions come from picturing the same thing: two copies of the same model marking each other's homework. That would deserve the nerves. It is also not what a serious automated audit looks like.

What an AI auditor actually is

In our system an auditor is a separate worker with one job: review the output of other workers against a written brief. The brief is not "check this looks good". It is a list of testable criteria. Do the figures in the report reconcile with the source data. Does every factual claim carry a source. Does the draft follow the style rules, including the ones about what never to say. Is the schema valid, are the links live, is the customer's name spelt the way the record spells it.

Alongside the criteria sit thresholds. Above this confidence, pass. Below that, fail with reasons. In between, escalate to a person. The auditor's own performance is tracked: humans sample its passes and fails on a schedule, and its error rate is a number on a dashboard, not a matter of faith. If it starts failing, a circuit breaker stops the line it supervises until someone investigates.

Where machine review is the right tool

Three conditions, and you want all three. The volume is high, so human review of everything would be the bottleneck that kills the economics. The criteria can be written down, because an auditor without testable criteria is just a vibe with a job title. And the blast radius is small: a wrongly passed item costs an edit or a correction, not a lawsuit or a lost client.

Content quality checks, data validation, reconciliation, formatting and compliance-against-checklist work all pass this test comfortably. That is most of the review volume in most operations, which is exactly why automating it matters.

Where a human signs, every time

Money leaving the business. Anything with legal exposure. Public statements in the company's name beyond routine templated content. Personnel decisions. Commitments to clients. New categories of work the system has not done before. In these lanes the human is not a formality to be optimised away. The human is the point, because these are the decisions where judgement, accountability and context beat any written criteria.

The design principle underneath all of this is that the choice between human and machine review is made per step, never per system. A workflow is not "human in the loop" or "fully autonomous". It is forty steps, each with its own risk class, and the review it deserves at each one. Get that granularity right and you get both things people assume are in tension: an operation fast enough to be worth building, and one careful enough to be allowed near the work that matters.