Case study · Document extraction
Three weeks to next day
A large home retailer set the bar at five centavos per invoice. That number is why the system does not rely on the model to know when it is right.
Allan Tan · Founder and Chief AI Scientist
Three hundred documents a day. Half of them on paper.
The home retailer at the centre of this project moves roughly 10,000 invoices and delivery receipts a month, and about half of those arrive the way they have always arrived: as physical documents, delivery receipts especially, creased and stamped and handed over at a loading bay.
Thirty people were employed to key them in and check them. Reconciliation ran three weeks behind.
The bar finance set
Most document-extraction projects begin with a conversation about accuracy in which everybody agrees it should be high. This one began with a number.
One hundred per cent accuracy is needed. Threshold of 0.05.
Five centavos, on the total of an invoice. Not an average across the run, not a per-field tolerance that a good month could absorb. One document, one number, five centavos.
It is worth sitting with how unforgiving that is, because it rules out an entire class of solution. A model that reports itself 94 per cent confident is telling you nothing useful against a five-centavo bar. Ninety-four per cent of ten thousand documents a month is six hundred documents wrong, and the finance team does not know which six hundred.
| Line item | 1,240.00 |
|---|---|
| Line item | 3,150.00 |
| Line item | 890.50 |
| Line item | 2,475.00 |
| Subtotal | 7,755.50 |
| VAT at 12% | 930.66 |
| Computed total | 8,686.16 |
| Total stated on the document | 8,686.23 |
| Out by | 0.07 |
Arithmetic, not confidence
So the system does not ask the model how sure it is.
Once a document has been read, the extracted values are checked against each other the way a bookkeeper would check them. Do the line items sum to the subtotal? Does the tax reconcile? Does the subtotal plus tax equal the stated total, to within five centavos?
If all aligned, we assume it has high confidence. If misaligned, candidate for manual check.
This is a deterministic check sitting on top of a probabilistic one, and the division of labour matters. The model is good at reading a smudged delivery receipt. It is not qualified to certify its own output to a finance department. Arithmetic is.
The consequence is that the system’s uncertainty is real rather than declared. A document is not flagged because a confidence score dipped below a threshold somebody chose; it is flagged because the numbers on it do not add up, which is a fact about the document and not an opinion about the model.
What the thirty people do now
Everything that reconciles goes through. Everything that does not lands in a queue for a human, with the specific mismatch shown.
That inverts the job. The team was previously keying in every document and checking a sample. It now checks only the documents the system could not stand behind, which is the work that actually needed a person, and considerably less of it.
The number that changed
Extraction accuracy before and after is not a figure this project can quote. The manual baseline was never measured in a way that would make the comparison honest, and inventing one would undermine the point of everything above.
What did change, unambiguously, is time.
3 weeks → next day
Reconciliation, before and after
For a retailer, that is not a productivity statistic. Three weeks behind means the business is making decisions on a picture of itself from three weeks ago: supplier disputes raised late, discrepancies found after the goods have sold, cash position always slightly historical. Next day means the books describe the business as it currently is.
The saved effort is real. The saved time is what the finance director noticed.