Rewrite or review? AI for document data, local and in the cloud

The problem

People in companies still retype data from documents into their systems: purchase orders, delivery notes, complaints, inspection reports. AI can prefill that data today and a person only reviews it. The questions our clients ask are practical: Is local AI (no data leaves the company) accurate enough? Can AI tell where it is unsure? Does reviewing really save time compared with retyping? And at what volume does your own hardware pay off?

What we built

An open benchmark on 60 fictional Slovak documents (4 types × 15: purchase orders, delivery notes and CMRs, complaint forms, electrical inspection reports; 5 clean, 5 scanned and 5 degraded each: faded, phone photo, skewed and blurred, fax). Each document has the correct answer field by field. Every model got the same prompt and the same output schema.

We wrote down and froze the questions, metrics and the bar for a “good result” before the first measurement (preregistration). We publish the results even if the local model loses.

How we measured

• Local: Qwen 3.6 (35B) and Gemma 4 (26B) via Ollama on a mini PC with an AMD Ryzen AI 9 HX 370 and 64 GB RAM, €1,474 for the build. Two paths: the page image straight into the model, or OCR first (GLM-OCR) and then text into the model.
• Cloud: Claude Sonnet 5.5 and Claude Haiku 5.5, page image.
• 60 documents × 3 repeats = 180 extractions per combination. Temperature 0 where the model accepts it.
• Scoring: text must match at least 90% and diacritics must be exact; dates, amounts, company IDs and numbers exactly; order line items row by row. An invented value where the document has none is the worst error and is counted separately.

What we measured

4 Nov 2026, 60 documents × 3 repeats per row.

• Accuracy: the best local model is 5.3 percentage points behind the best cloud model. It narrowly missed our preset bar (at most 5 points).
• Gemma 4 is almost as accurate as Qwen and 35% faster, so the cheaper machine configuration is enough.
• Image straight into the model beats OCR plus text by 3 to 6 points.
• Repeats differed by at most 1.1 points; results are stable.
• All numbers include one correction of an error in our dataset (see below). Under the original rules: Sonnet 98.9%, Haiku 97.2%, Qwen 93.6%. The gap between the best local and best cloud model is 5.3 points either way.

Does AI know when it is wrong?

The model could mark any field “check this”. We wanted it to mark at least 80% of its own errors while marking no more than 20% of all fields.

No model met the bar. Local models on images mark only 5 to 10% of their own errors. Simple rules in code (valid date, company ID checksum, line items add up to the total) add another 5 to 12 points. Anyone deploying AI on documents can’t rely on the AI to point out where it went wrong.

What it costs

Estimate: we did not measure power with a meter; we assume 65 W under load (upper bound 120 W, the adapter rating), 12 W idle and €0.1985/kWh. Cloud cost is actually billed for 180 documents. €1 = 25 CZK, $1 = €0.86.

• The local machine is 99% hardware depreciation (€1,474 over 3 years). Electricity per document is negligible (€0.0001).
• Local pays off against a 1 CZK SaaS from ~1,070 documents a month, against Sonnet from ~3,500. Against Haiku only from ~60,000, which is close to one machine’s capacity (~90,000 documents a month on a single stream).
• At small volumes the main reason for local AI is that data never leaves the company, not price.
• The biggest cost in every case is the person’s review time. We measure that in the study below.

Human study: in progress

Two questions remain that need people:

1. Is reviewing prefilled data at least 2.5 times faster than retyping it?
2. Do people notice errors in prefilled data, and do “check this” marks help them?

Join the study: 16 fictional documents in the browser, about 45 minutes, a nickname is enough. The study is in Slovak. We will add the results to this page.

What we learned

• An error in our own dataset. For complaints without a resolution, the generator printed “to be decided within 3 working days” on the document. Both cloud models faithfully copied it and our scoring counted it as an invented value. We recorded the fix as a dated amendment to the preregistration and report both numbers.
• A gap in the instructions looks like a model error. On inspection reports, local models often appended the address to the object name. Cloud models did not. The prompt didn’t forbid it, so after the freeze we left it and count it as an error.
• “As a number” in the prompt with a text field in the schema produced values like “,4” and whole empty table rows under Ollama. Writing “digits only” fixed it.
• OCR loses checkboxes and diacritics on degraded documents, which is why the image path is more accurate.
• The runtime version changes results: after an Ollama update, Gemma’s accuracy moved by 3.7 points with the same weights. Record versions.

Data and code

The dataset (60 documents with correct answers, CC BY 4.0), the generator, the preregistration, the measurement code and raw outputs will be published together with the study results.

Model names only identify what we measured. LabZone is not affiliated with the model makers, and they did not review or endorse these results.

Who it is for

Companies that retype data from documents by hand and are considering AI, but don’t know whether a local model is enough, what it will cost and how much work it really saves a person.

We store questions without IP addresses for 30 days to evaluate and improve the experiment. We use your IP address only briefly to limit the number of requests and do not store it. Please don't include personal data in your questions.