Methodology
What was measured, on what data, against whom, under which conventions — and the six places where this evaluation is weaker than it looks.
A benchmark result is only as useful as the account of how it was produced. This page is that account. It is deliberately unflattering in places, because the alternative — a results table with no methodology behind it — is what makes most Arabic OCR accuracy claims impossible to evaluate.
The benchmark
KITAB-Bench (Heakl et al., Findings of ACL 2025) contains 8,809 samples across nine tasks and 36 sub-domains. It was built and peer-reviewed by a group unaffiliated with YaiGlobal. We did not design it, contribute to it, or influence its composition.
Task selection, and what was excluded
Four of nine tasks were evaluated. The five exclusions were made on capability grounds, and stated before results were produced:
- Chart-to-DataFrame, diagram-to-JSON and visual question answering measure semantic reasoning over a document rather than faithful extraction from it. They are a different capability.
- Line detection and line recognition measure an internal representation the parser does not expose.
These are reasoned exclusions, but they are still exclusions. No claim from this evaluation speaks to chart interpretation, diagram understanding, document question answering, or line-level detection. A reader comparing this report against a full-suite KITAB-Bench evaluation of another system is comparing different things.
| Task | Samples | Pool coverage | Baselines |
|---|---|---|---|
| Image-to-text recognition | 3,760 | Complete | 17 |
| Layout detection | 1,700 | Arabic BCE-Arabic-v1 pool; 400 English DocLayNet pages excluded | 6 |
| Table structure recognition | 456 | Complete | 14 |
| End-to-end PDF-to-Markdown | 33 | Complete | 11 |
Layout detection excludes the English DocLayNet pages because the subject of the study is Arabic document processing and because the published baselines report BCE-Arabic and DocLayNet separately. The comparison is therefore drawn against the baselines' BCE-Arabic rows, which keeps it on matched data.
Baseline provenance — the most important caveat
The great majority of baseline figures are taken from the published KITAB-Bench paper, Tables 4, 5, 6 and 9, produced by that paper's authors using each system's official implementation or API endpoint. We did not re-run them.
This has one advantage and one disadvantage. The advantage is that we compare against figures we had no opportunity to influence. The disadvantage is that differences in harness, prompt, pre-processing or model version are confounded with genuine capability differences.
Two baselines are exceptions, and this matters. LlamaParse and Landing AI do not appear anywhere in the published KITAB-Bench results; their figures throughout were measured in our own run. LlamaParse is the strongest baseline on three of the four evaluated tasks — table recognition, PDF-to-Markdown and layout F1 — so several headline comparisons rest on internally measured numbers rather than independently published ones.
The practical consequence: until independent replication exists, those three comparisons should carry less weight than the image-to-text comparison, which is drawn entirely against published figures.
Metrics and conventions
| Task | Primary | Auxiliary |
|---|---|---|
| Image-to-text | CER ↓, WER ↓ | chrF ↑, BLEU ↑, METEOR ↑ |
| Table structure | TEDS ↑ | — |
| PDF-to-Markdown | MARS ↑ | chrF ↑ (text), TEDS ↑ (tables) |
| Layout detection | F1 ↑, mAP@0.5 ↑ | Precision ↑, recall ↑, mAP@0.75, mAP@0.5:0.95 |
Error rates are expressed as percentages
Character and word error rates are normalized edit distances, conventionally reported as rates. Throughout, we express them as percentages: a CER of 0.125 is reported as 12.50%. This is a change of presentation only, applied uniformly to our results and to every baseline figure quoted, so all comparisons remain on a common basis.
A caution about citation. Because the underlying quantity is a ratio, writing this figure as 0.125% overstates accuracy by a factor of 100. The correct forms are a CER of 0.125 or a CER of 12.50%.
These rates are not bounded above by 100%. Edit distance exceeds the reference length when a system inserts spurious content, and several baselines exhibit this — Surya reaching a 495.00% CER. Such values are real measurements, not artefacts.
Because the quantity is already a percentage, we distinguish throughout between a difference in percentage points and a relative difference in percent.
Macro averaging
Aggregate image-to-text figures are unweighted macro averages across twelve subsets, following the published KITAB-Bench convention. This gives a 10-sample subset the same weight as an 800-sample subset, so it does not estimate expected performance on a randomly drawn page.
We follow the convention because the published baselines do and comparison requires it. It is not neutral in our favour: the sample-weighted alternative is worse for our system, a 14.08% CER against 12.50%.
A discrepancy in the official sample count
One inconsistency in the source benchmark should be recorded. The published dataset distribution lists KHATT twice under image-to-text — a 200-sample handwritten-paragraph split and a 200-sample handwritten-line split — and totals the task at 3,760 samples. The official per-dataset results table reports a single KHATT row, and its twelve listed subsets sum to 3,560.
We follow the official convention: the twelve subset rows are reported as measured, and the task aggregate is labelled 3,760 as in the source. The difference does not affect any macro average, since macro averaging is over subsets rather than samples.
Six limitations
1. Baseline provenance
Most baseline figures are quoted rather than re-run, so prompting, pre-processing, model version and API behaviour are confounded with capability. More seriously, LlamaParse and Landing AI were measured internally, and LlamaParse is the strongest baseline on three of four tasks. Independent replication would materially strengthen these claims, and until it exists those three comparisons should carry less weight.
2. Five of nine tasks are not evaluated
The exclusions are reasoned, but they are exclusions. No claim here speaks to chart interpretation, diagram understanding, document question answering, or line-level detection and recognition.
3. Small evaluation sets in places
PDF-to-Markdown uses thirty-three documents; the HistoricalBooks subset ten samples; the footnote layout category fifty-four instances; the table layout category twenty-eight. Differences computed on these are not statistically robust. We report no confidence intervals because per-sample scores for published baselines are not available. Small-set figures should be read as indicative.
4. Macro averaging
As above: the convention does not estimate expected performance on a random page, and the sample-weighted alternative is worse for us.
5. Benchmark composition
KITAB-Bench itself notes that it under-represents low-resource dialects and institutional scans such as governmental and financial records, and that a significant portion of its table, chart and diagram material is synthetically generated rather than naturally occurring. Strong performance here does not automatically transfer to those under-covered domains.
6. Possible train-test contamination in baselines
AIN-7B's reported 0.00% CER on ADAB is difficult to explain other than by overlap between its training data and the benchmark. We have no visibility into the training corpora of any baseline. Where such a result inflates a baseline, our reported margin is understated; we flag it rather than rely on it.
What can and cannot be claimed from this
The defensible statement is bounded, and we would rather state it precisely than broadly:
"In a vendor-conducted evaluation on KITAB-Bench, YaiGlobal Parser recorded the lowest character error rate of the seventeen systems compared on the image-to-text track."
What cannot be claimed: that YaiGlobal is the most accurate Arabic OCR without qualification; that the result has been independently verified; or that it speaks to the five tasks not evaluated. No independent party has yet replicated this evaluation. If you are in a position to attempt it, we would rather you did than didn't — get in touch.
Source. Heakl, A., Sohail, A., Ranjan, M., Hossam, R., Ahmad, G. S., El-Geish, M., Maher, O., Shen, Z., Khan, F., and Khan, S. (2025). KITAB-Bench: A Comprehensive Multi-Domain Benchmark for Arabic OCR and Document Understanding. Findings of the Association for Computational Linguistics: ACL 2025. arXiv:2502.14949. DOI 10.18653/v1/2025.findings-acl.1135.