Benchmark

Benchmark: Doxtract on messy real-world documents

Clean, digital PDFs are easy. Most models get them right. The documents that break extraction are the ones people actually have: receipts photographed on a phone, pages scanned sideways, and old office scans.

We ran Doxtract and a small general-purpose LLM with one plain prompt on the same 50 documents, using the same JSON schema for both. On clean multi-page PDFs the two tie. On messy ones, the plain prompt falls apart.

Last updated . Long-document results added 10 Nov 2026.

Messy documents

Where a plain prompt falls apart

Document typeDoxtract, fields correctPlain LLM prompt, fields correct
Rotated pages100% (28/28)39.3% (11/28)
Phone photos96.2% (75/78)56.4% (44/78)
Scanned receipts (SROIE)100% (60/60)73.3% (44/60)

On phone photos the plain prompt also made up 11.8 values per 100 fields, against 0.0 for doxtract. It missed 73.1% of their line items, against 3.8% for doxtract.

Headline

Headline results

Results on the 45 documents whose sources allow us to publish results.

Metric (45 documents)DoxtractPlain LLM prompt
Fields correct98.7% (296 of 300)69.7% (209 of 300)
Valid JSON100% (45 of 45)100% (45 of 45)

Full results

Every metric, 45 published documents

Metric (45 documents)DoxtractPlain LLM prompt
Fields correct98.7% (296 of 300)69.7% (209 of 300)
Made-up values per 100 fields0.0 (0 of 1,272)1.5 (18 of 1,212)
Line items missed0.4% (2 of 526)9.1% (48 of 526)
Extra line items added122
Valid JSON100% (45 of 45)100% (45 of 45)
Median latency per document7.6 s4.7 s

Totals for all 50 documents, including the 5 old scans we score but do not publish (see the caveats):

Metric (all 50 documents)DoxtractPlain LLM prompt
Fields correct98.5% (333 of 338)70.1% (237 of 338)
Made-up values per 100 fields0.1 (1 of 1,330)1.9 (24 of 1,252)
Line items missed2.9% (16 of 550)12.9% (71 of 550)
Valid JSON100% (50 of 50)100% (50 of 50)
Median latency per document7.5 s4.2 s
Per-document results (CSV, 45 documents)
  • system Doxtract or plain_prompt
  • fields_scored, fields_correct header fields in the answer key, and how many matched
  • made_up_values, made_up_checks made-up values, and how many returned values were checked for them
  • items_in_key, items_matched, items_extra line items in the answer key, matched, and added with no match
  • wrong_fields the header fields that did not match, separated by semicolons

By document type

Results by document type

SliceDoxtract, fields correctPlain LLM prompt, fields correctNote
Rotated pages100% (28/28)39.3% (11/28)
Phone photos96.2% (75/78)56.4% (44/78)Made-up values per 100 fields: 0.0 vs 11.8. Line items missed: 3.8% vs 73.1%
Receipts, phone photos (CORD)96.7% (87/90)58.9% (53/90)
Receipts, flatbed scans (SROIE)100% (60/60)73.3% (44/60)
Invoices (Mendeley samples)98.8% (79/80)52.5% (42/80)
Bank statements (synthetic)100% (30/30)100% (30/30)Tie

Long documents

Every row of a long document

A long table is where a single model call quietly drops rows. These are four long documents we generated, so the answer key is exact. One run each, citations on.

DocumentPagesRows returnedFields correctTime
Invoice, dense table3150 of 15099.8%19 s
Invoice, very long281,650 of 1,650100%71 s
Bank statement5250 of 250100%31 s
Bank statement, scanned5250 of 25099.9%23 s

Added . Fields correct counts header fields and every row.

  • What changed. An earlier build of ours returned 1,528 of the 1,650 rows on the 28-page invoice and gave no warning: pages with many short rows were sent two to a call and the model skipped some. Pages are now sized by how many table rows they hold, so a page like that goes in a call of its own. The same file now returns every row.
  • Thin answers are retried. After the first pass, a chunk that returns far fewer rows than the chunks around it is read once more with a note saying how many table lines its pages print. The fuller answer is kept, and never one with more rows than the pages print.
  • Citations. With citations on, each value points to where it was read. On the 28-page invoice 4,399 of 6,111 cited values were confirmed against the printed text, and none of the confirmed ones were wrong. Scanned pages have no text layer, so nothing on them is confirmed.

Honest fit

Where the baseline wins or ties

  • Speed. The plain prompt is faster: a median of 4.7 seconds per document against 7.6 seconds for Doxtract on the 45 published documents (4.2 s against 7.5 s across all 50). If speed matters more than accuracy on messy files, that is a real trade-off.
  • Clean multi-page PDFs. Both scored 100% on the synthetic bank statements and long invoices (70 of 70 fields each).
  • Valid JSON. Both returned valid, schema-shaped JSON on every document. Valid JSON is not the same as correct values.

In practice

What this means in practice

If your documents are clean digital PDFs, a plain prompt to a small model may be enough. If they come from phones, scanners or email attachments of unknown quality, the plain prompt gets fields wrong or invents them, and it gives you no sign that anything went wrong. That is the case Doxtract is built for.

Method

How we ran it

  • Documents. 50 documents. Public datasets: 15 CORD phone-photo receipts, 10 SROIE scanned receipts, 10 Mendeley sample invoices and 5 RVL-CDIP old office scans (invoice class). We added 5 long synthetic invoices and 5 multi-page bank statements we generated, where the generator input is the answer key. 4 images were rotated for the rotated-page test (2 CORD receipts and 2 Mendeley invoices, by 90 or 180 degrees).
  • Published results. 45 of the 50 documents come from sources that allow published results. The 5 RVL-CDIP scans are scored but not published, so every number on this page uses the 45, except the all-50 totals, which are labelled.
  • Answer keys. Built from the dataset labels and completed by hand, then frozen before either system ran. A person spot-checked them against the original images. A field that is not printed on the document is recorded as empty, which is how made-up values are caught.
  • Same schema for both. One JSON schema per document type (receipt, invoice, statement), sent unchanged to both systems.
  • Baseline. A small general-purpose LLM (GPT-5 nano, minimal reasoning) with one plain prompt: extract these fields and return only JSON matching this schema. JSON mode on, no examples, no retries for quality, no pre-processing. This is roughly what a developer would wire up in an afternoon.
  • One run each. Each document went through each system once. Only transport failures with no output (rate limits, server errors, timeouts) could be retried.

Scoring

  • Exact field match after normalization: dates in ISO format, amounts as numbers within 0.01, currency symbols mapped to ISO codes, other text trimmed and case-folded.
  • Names (merchant, vendor and account holder) match on token overlap, with a token F1 of 0.8 or more. There are no address fields in the schemas.
  • An empty field matched by an empty answer counts as correct.
  • A line item matches when its description is at least 0.8 similar and its amount is within 0.01. A missing quantity counts as 1.
  • Statement transactions match on absolute amount, and the sign is scored separately.
  • A document that does not return valid JSON scores 0 for every field.
  • Made-up value: a value returned for a field that is empty on the document, or a wrong number or date that appears nowhere on the document.
  • Line item missed: an item in the answer key with no matching item in the output.
  • Extra line item: an item in the output with no match in the answer key.
  • Valid JSON: the output parses and validates against the schema, where any single value may be null.

Caveats

What this does not show

  • Long documents: one run, our own files, no baseline. The long-document table is one run on four documents we generated, so it shows the pipeline works at length, not how it does on yours. We did not run the plain prompt on these files, so there is no comparison. Real long invoices are harder: on real broadcast invoices that list weekly lines under each contract line we still add or lose rows, and the same file can return a different row count from one run to the next. Totals were stable.
  • Small sample. 50 documents is enough to show a pattern, not to give precise rates. A difference of a few points on one slice can come from one or two documents, and with one run each, a single field can flip between runs.
  • Old scans are not published. We also score 5 low-resolution invoice scans from RVL-CDIP, but we do not publish their per-document results or images. They only appear in the all-50 totals. On one of them Doxtract read every description but shifted the amounts by one row, so 14 of the 16 line items it missed across all 50 documents come from that one scan. Very old, low-resolution scans are still the hardest case.
  • Print-and-photograph test not done yet. We plan to print some documents and photograph them at an angle. That variant is not in these results.
  • One baseline. We tested one small model with one plain prompt. A larger model, or a tuned prompt with retries and validation, would likely score differently. We have not tested those.
  • Synthetic documents. The invoices and statements we generated are cleaner and more regular than many real ones.

Credits

Dataset credits

  • CORD. Consolidated Receipt Dataset, NAVER Clova. Licensed CC BY 4.0. Source
  • SROIE. ICDAR 2019 Robust Reading Challenge on Scanned Receipts OCR and Information Extraction. Licensed CC BY 4.0. Source
  • Samples of electronic invoices. Mendeley Data, doi 10.17632/tnj49gpmtz.2. Licensed CC BY 4.0. We used the copy published on Hugging Face as katanaml-org/invoices-donut-data-v1. Source
  • RVL-CDIP. A subset of the IIT-CDIP collection, drawn from the Legacy Tobacco Documents Library (tobacco industry litigation archive). Scored internally, not shown. Source
  • Changes made. Some images were rotated (2 CORD receipts and 2 Mendeley invoices) and re-saved. Labels were normalized and completed by hand for the benchmark schemas.

FAQ

Benchmark questions, answered.

What is the baseline?

GPT-5 nano with minimal reasoning, given one plain prompt and the same JSON schema as Doxtract, with JSON mode on. No examples, retries or pre-processing.

Why compare with a small model and a plain prompt?

It is the cheapest and quickest thing a developer can build. The question this benchmark answers is whether that is good enough for real documents. On clean multi-page PDFs it was. On phone photos, rotated pages and scans it was not.

Is Doxtract faster?

No. The baseline was faster, with a median of 4.7 seconds per document against 7.6 seconds for Doxtract on the 45 published documents.

What counts as a made-up value?

A value returned for a field that is not on the document, or a wrong number or date that appears nowhere on it. Doxtract returns null for missing fields instead of guessing.

Both returned valid JSON. Why does accuracy differ?

Valid JSON only means the output has the right shape. The values inside can still be wrong, missing or invented. That is what the fields correct and made-up values columns measure.

How many documents were tested?

50 in total. The numbers on this page use the 45 whose sources allow published results. The other 5 are old invoice scans from RVL-CDIP that we score but do not publish.

How does Doxtract handle long documents?

It splits the document into chunks sized by how dense each page is, reads them in parallel, joins the rows in page order and checks the result. A chunk that comes back thin is read once more. If a chunk fails, you get the pages that worked, the failed page ranges and a warning, not an error for the whole file. On our four long test documents it returned every row.

Can I test my own documents?

Yes. Upload one in the Playground or use a converter for invoices, receipts and bank statements.

Will the results be updated?

Yes, when the print-and-photograph test is added.