Shadow mode: putting an agent into production without betting the business on it

Shadow mode is a simple idea with an outsized effect on whether automation survives contact with reality. The automated system runs on live inputs, exactly as it would in production. It reads the real invoices, drafts the real replies, produces the real reports. But nothing it produces is released. A person still does the job, and the system's output sits alongside theirs, checked and scored.

For a few weeks, the business pays for the work twice. That sounds wasteful. It is the cheapest insurance available.

What shadow mode tells you

First, accuracy against the humans, on your data rather than a vendor's demo set. Not "the model is very capable" but "on our last four hundred cases it matched or beat the team on 96 per cent and here are the sixteen it missed". Second, the shape of the exceptions. Every process has cases nobody mentioned in the mapping workshop, and shadow mode surfaces them while they are harmless. Third, throughput and cost per task, measured rather than projected. Fourth, whether the review gates are placed correctly, because the person checking the output quickly learns which categories they never need to look at and which they always do.

Running it properly

Set the exit criteria before you start. Something like: two consecutive weeks at or above the human accuracy baseline, no unexplained failures, cost per task under the agreed figure. Writing the bar down in advance stops the two bad endings, the one where an impatient sponsor declares victory after four good days, and the one where a nervous team keeps extending the trial forever because no number was ever going to feel safe.

Give it long enough to see variety. A fortnight of Tuesdays does not include month-end, the holiday backlog, or the supplier who formats everything strangely. Weeks, not days.

Then cut over gradually, by risk class rather than all at once. Let the system release the low-stakes categories first while the sensitive ones stay gated. Keep the rollback path warm: if the numbers move the wrong way after cutover, the previous way of working should be one decision away, not a rebuild away.

Why teams skip it, and what it costs them

Teams skip shadow mode for understandable reasons. The demo went well, the quarter is ending, and running two systems feels like paying twice for one job. But the pilots that go straight from demo to production are the ones that fail in public, and a system that fails in public does not get a second chance, whatever its error rate improves to later. Shadow mode is how an automated workflow earns the right to be trusted, with evidence the finance director can read.