Intelligent document processing: test it on your own documents first

Vendor demos use clean documents. Yours aren't. Intelligent document processing reads incoming invoices, delivery notes, forms and contracts, pulls out the fields you need and sends them to your systems, with a person checking anything the system isn't sure of. It replaces manual keying: at 1,000 documents a month, our illustrative example releases about 49 hours a month.

So test it on your own post first.

Intelligent document processing at a glance

AI document processing is a pipeline of steps around a model, and a person stays on the ones that need judgement.

What IDP does, in four steps

PDF data extraction is one of four steps:

1. Classify. What kind of document is this?
2. Extract. Which fields does it hold, including tables and line items?
3. Validate. Do the values make sense against your rules and records?
4. Route. Into the target system, or to a person.

A text PDF reads as it is; scans and photos need optical character recognition (OCR) or a vision model. Tables are the hard part.

Paper hasn't gone. A 2025 AIIM survey of enterprises in the US and German-speaking Europe, sponsored by IDP vendors, found 61% of IDP processes still include paper.

Extraction underpins most systems in our Insights guides. For invoice VAT fields, posting and duplicates, see automated invoice processing.

LLM extraction vs template OCR: where each one wins

AI data extraction with a large language model (LLM) isn't always the right tool. Independent tests split on how messy the documents are.

LLMs win on messy input. On scanned receipts, Berghaus and colleagues found a vision model got 87.46% of fields on scanned receipts, against no more than 47.00% when the page was converted to text first.

Templates win on volume. Scored on F1, where 1.0 is perfect, Wang and Shen's table-based pipeline reached an F1 of 1.0 in 0.97 seconds on structured documents, where direct LLM extraction needed 13.4 to 13.6 seconds each.

Hand-coded patterns are a poor fallback. On synthetic cheques, Patel and colleagues found an OCR plus regular-expression baseline scored an F1 score of 0.395. The same study showed how LLMs fail: by over-extracting. One model returned 686 false dates against 304 correct ones. A confidence score wouldn't catch that. A rule check would.

So most SME set-ups are hybrids: parse what's structured, template the high-volume fixed layouts, send the rest to an LLM. It's the case for mixing rules and agents.

How to test extraction accuracy on your own documents

No sales demo covers this. Run these eight steps, in order, on any tool you're weighing up.

1. Build a test set from real documents. Use recent documents in the mix you actually receive: every type, the worst scans, phone photos, the odd supplier. Don't let the vendor choose.
2. Size it with the rule of three. If a system makes no errors on n documents, you can say with 19-in-20 confidence that its true error rate is at most 3 in n (Hanley and Lippman-Hand, 1983). So a clean run on about 300 documents shows an error rate under one in 100.
3. Label the right answers first. Before the system runs, a person keys the correct value for every field. That's your ground truth.
4. Measure at field level, not document level. Score every field (supplier, date, total, VAT number, each table row) as right, wrong or missing. A strong document-level score can hide a field that's wrong one time in three.
5. Set a confidence threshold per field. Above it, the field goes straight through; below it, a person checks. At each threshold, record the straight-through rate and the number of errors that slipped through above it. The second number is the one that matters.
6. Add rule checks. Totals add up, dates fall in a sensible range, the supplier or customer exists, the VAT number has a valid format. Rules catch the confident mistakes.
7. Run shadow mode on live documents. Give it two to four weeks alongside your team, releasing nothing. Our shadow mode note shows how.
8. Agree go-live criteria before you start. For example: accuracy at or above your team's on the same documents, no errors above threshold on critical fields, and a straight-through rate that pays its way.

The UK government worked this way. i.AI tested Extract on over 400 existing digital records from over 30 planning authorities, compared its output directly with records made by hand and set accuracy thresholds in advance. Outputs needed only minor edits in around two-thirds (59 to 72%) of cases.

Handling mixed document types

One inbox can carry invoices, credit notes, statements, delivery notes and remittances. Test classification on its own: a misfiled document gets the wrong fields.

Give each document family its own model or prompt and threshold. Split multi-document PDFs, such as five invoices in one scan, before extraction, and test the split too. Statements are a document type of their own, covered in supplier statement reconciliation. Extracted delivery notes feed three-way matching.

Keying time and cost at 1,000 documents a month

Data entry automation is where the hours show. Take a business keying 1,000 mixed documents a month (illustrative, not a client result). The only measured public benchmark covers planning records. So every input except pay is our assumption, and pay is the median UK full-time hourly pay plus employer National Insurance and pension.

One cross-check. i.AI measured review and correction at 5 to 20 minutes per planning record, against our 2 minutes per document, because a typical business document has far fewer fields. Shadow mode will give you your own figure.

Released hours aren't cash unless someone uses them. Data-entry staff are likely paid below the median too, so treat the value as an upper estimate.

Priced in dirhams, those hours are worth about AED 2,831 a month at our UAE rate of about AED 57 an hour. No official UAE pay data by occupation exists, so the rate rests on an assumed package of AED 8,000 a month and gratuity at 21 days' basic pay for each of the first five years. For Emirati staff the private-sector floor is AED 6,000 a month.

Document extraction benchmarks

No vendor figures here. Academic results use public or synthetic datasets, not yours, and arXiv papers are preprints.

Choosing IDP software: build, buy or configure

• Buy a packaged tool when your documents are common types (invoices, receipts) and your target system has an integration.
• Configure a general model with your own fields and rules for many document types at moderate volumes.
• Build when documents are unusual, rules are yours alone, or the output lands in a bespoke system.

The usual build-or-buy question applies: bespoke or off the shelf. Whichever route you take, run every candidate through the eight-step test plan on the same test set.

How Orwell builds document processing

Our business automation work on documents runs in six stages, test plan first.

1. Map. We map the documents and where each field goes.
2. Test. With your team, we build the labelled test set and run the eight steps.
3. Shadow. It runs on live documents, releasing nothing, until it meets the go-live criteria.
4. Gate. Low-confidence or rule-breaking fields stop at an approval gate.
5. Go live by type. Highest-volume document type first. We measure cost per document before and after.
6. Monitor. A weekly sample of straight-through documents is checked by a person; drift alerts fire when a layout changes.

What document processing won't do

• It won't be perfect, and doesn't have to be. What counts is catching and correcting errors.
• It won't read what isn't legible. Bad scans and difficult handwriting go to a person.
• It won't catch a confident wrong answer on its own. That takes rule checks.
• It won't stay accurate unwatched. Layouts change; accuracy drifts.
• It won't make decisions about people. Anything with a significant effect on a person keeps a human on it.
• It won't handle Arabic PDFs reliably yet. In KITAB-Bench the best model managed only 65% accuracy on Arabic PDF-to-Markdown conversion.

Gartner's forecast is that over 40% of agentic AI projects will be cancelled by the end of 2027. Test before you commit.

UK and UAE rules for extracted data

IDP itself isn't regulated; the rules attach to the data and what it's used for.

Common questions

Tell us which documents your team keys in today, where the data ends up and roughly how many arrive each month, and bring a few real examples to the call, especially the awkward ones. After an informal scoping chat we'll send a price range for automating the extraction. Document mixes, volumes and target systems differ too much for a published price to be honest. Select "Business automation" on the form.