Arabic OCR, measured on KITAB-Bench
Four tasks. 5,949 samples. Seventeen baselines. Ten of thirteen headline measures led — and the three that were not are on this page too.
Most Arabic OCR accuracy claims are unfalsifiable. They cite no benchmark, no sample count, no comparison set, and no definition of the metric being claimed. This page is our attempt to do the opposite: state exactly what was measured, on what data, against whom, and where we lose.
What KITAB-Bench is
KITAB-Bench is a peer-reviewed Arabic OCR and document-understanding benchmark, published by Heakl and colleagues in the Findings of ACL 2025. It contains 8,809 samples across nine tasks and 36 sub-domains, spanning printed text, handwriting, tables, charts, diagrams, layout analysis and end-to-end PDF conversion.
It matters because it is not ours. Arabic OCR has no shortage of vendor-defined evaluations in which the vendor wins; a benchmark built and peer-reviewed by an unaffiliated research group is a materially different object. We ran our system on it and report the result against the field.
This is a vendor-conducted evaluation. YaiGlobal ran YaiGlobal's system and compared it against baseline figures, most quoted from the published KITAB-Bench paper and two measured internally. No independent party has replicated it. That is a real limitation, and it is why the methodology page exists.
What was evaluated, and what was not
Four of the nine tasks. Three of the four were scored on their complete KITAB-Bench pool.
| Task | Samples | Pool | Baselines |
|---|---|---|---|
| Image-to-text | 3,760 | Complete | 17 |
| Layout detection | 1,700 | Arabic BCE-Arabic-v1 portion of 2,100 | 6 |
| Table recognition | 456 | Complete | 14 |
| PDF-to-Markdown | 33 | Complete | 11 |
| Total | 5,949 | 67.5% of the 8,809-sample benchmark | |
Layout detection was scored on 1,700 of 2,100 samples: the full Arabic BCE-Arabic-v1 pool, excluding 400 English DocLayNet pages, because the subject of the study is Arabic document processing and the published baselines report the two pools separately. The comparison is therefore drawn against the baselines' BCE-Arabic figures, on matched data.
Five of the nine tasks were not evaluated — chart, diagram and visual-question tasks among them. Any claim we make is bounded by that.
Headline results
| # | System | Family | CER ↓ | WER ↓ | chrF ↑ |
|---|---|---|---|---|---|
| 1 | YaiGlobal Parser | Ours | 12.50% | 25.00% | 85.23 |
| 2 | Gemini 2.0 Flash | Closed | 13.00% | 32.00% | 77.95 |
| 3 | AIN-7B | Open | 20.00% | 28.00% | 78.33 |
| 4 | GPT-4o | Closed | 31.00% | 55.00% | 61.01 |
| 5 | GPT-4o-mini | Closed | 43.00% | 71.00% | 47.21 |
| 6 | Azure Document Intelligence OCR | Closed | 52.00% | 69.00% | 50.97 |
| 7 | Tesseract | Framework | 54.00% | 84.00% | 39.62 |
| 8 | EasyOCR | Framework | 58.00% | 89.00% | 45.47 |
| 9 | LlamaParse | Closed | 69.00% | 83.00% | 48.35 |
| 10 | PaddleOCR | Framework | 79.00% | 102.00% | 16.73 |
| 11 | Gemma3 | Open | 105.00% | 145.00% | 30.02 |
Eleven of seventeen shown. The mean CER across the baselines is 126.19% and the median 74.00%; only three systems achieve a CER below 31%. Arabic image-to-text is not a task most current systems perform adequately, and the small size of the competitive group at the top should temper any reading of this as a broad consensus.
Error rates above 100% are real measurements, not typos. Character and word error rates are normalized edit distances — they exceed the reference length when a system inserts content that was never on the page. One baseline, Surya, records a 495.00% CER. See the glossary.
Where a baseline leads
Three of thirteen headline measures. We report them because a result set with no losses in it is not a measurement.
| Measure | YaiGlobal | Leader | Leader | Gap |
|---|---|---|---|---|
| PDF-to-Markdown text fidelity (chrF) | 79.81 | LlamaParse | 82.51 | −3.27% |
| Layout localization (mAP@0.5) | 0.611 | DETR (Docling) | 0.750 | −18.53% |
| Layout recall | 0.711 | DETR (Docling) | 0.725 | −1.93% |
The layout result has a specific and diagnosable shape. Precision is 0.916 and recall 0.711 — the parser is consistently right about what it detects and inconsistently complete about what it detects. Captions are the clear weak point: precision 0.901 against recall 0.428 over 1,230 instances, meaning roughly six captions in ten are not detected as captions. The practical consequence is narrow but real — an undetected caption is not lost text, it is absorbed into the adjacent body region, but its association with the figure it describes is lost, which degrades figure-aware retrieval specifically.
The margin widens as the material gets harder
On the handwritten subsets the gap is more than eight times what it is on the aggregate task.
| System | CER ↓ | WER ↓ | Δ CER |
|---|---|---|---|
| YaiGlobal Parser | 14.50% | 37.25% | — |
| Gemini 2.0 Flash | 22.00% | 53.50% | −34.09% |
| AIN-7B | 26.00% | 47.25% | −44.23% |
| GPT-4o | 40.00% | 75.00% | −63.75% |
| Tesseract | 75.75% | 115.50% | −80.86% |
| Azure Document Intelligence OCR | 76.50% | 89.50% | −81.05% |
The generalization gap among baselines is the more striking feature. Azure OCR degrades from 52.00% CER on the full task to 76.50% on handwriting; Tesseract from 54.00% to 75.75%; Qwen2.5-VL-7B from 120.00% to 160.75%. Systems tuned for printed Arabic do not transfer. The parser moves in the opposite direction relative to the field, which indicates that its handwriting capability is trained rather than a by-product of general text recognition.
More on Arabic handwriting recognition →
Structure: tables and layout
On table structure recognition the parser records TEDS 97.99 across 456 samples, the highest of fourteen systems. The nearest competitor is LlamaParse at 90.55, then GPT-4o at 85.76 and Gemini 2.0 Flash at 83.08. Every one of thirteen table types scored above 93.
The ordering by family is clean — purpose-built parsers lead, general-purpose closed models follow, open-weight models trail them, and OCR-plus-pipeline combinations trail everything. This reverses the image-to-text ordering, where LlamaParse and Landing AI performed poorly at raw transcription. Table structure rewards an explicit geometric model of the page, which parsers have and which general models must reconstruct from pixels each time.
On end-to-end PDF-to-Markdown the parser records MARS 79.97 across 33 manually annotated PDFs — the highest of eleven systems — with table fidelity TEDS 80.12 and text fidelity chrF 79.81, the last of which trails LlamaParse. The evaluation set is small, and the methodology page notes what that means for statistical confidence.
Send us your hardest page
Benchmarks are averages over someone else's documents. The question that matters is how a system behaves on yours — the water-damaged page, the marginalia, the table that spans a fold. Send the worst example you have and we will run it and show you the output.
Scope of these figures. Vendor-conducted evaluation of YaiGlobal Parser on four of the nine KITAB-Bench tasks, 5,949 samples, against seventeen baselines. Image-to-text aggregates are unweighted macro averages across twelve subsets, following the published KITAB-Bench convention; the sample-weighted alternative for CER is 14.08%. Benchmark: Heakl et al., KITAB-Bench: A Comprehensive Multi-Domain Benchmark for Arabic OCR and Document Understanding, Findings of ACL 2025, arXiv:2502.14949, DOI 10.18653/v1/2025.findings-acl.1135.