Full results

Every task, every baseline, every measure — including the three where a competitor leads.

These are the complete per-task tables. Read them alongside the methodology, which explains where each baseline figure came from and why three of the four task comparisons should carry less weight than the first.

1. Image-to-text recognition

3,760 samples · 17 systems · twelve corpora. CER and WER are lower-is-better; chrF is higher-is-better. Δ columns are relative differences against YaiGlobal.

2. Handwriting

Four handwritten corpora: KHATT, ADAB, Muharaf and OnlineKhatt. These are included in the image-to-text aggregate above, as they are in the published baselines, but analysed separately because they exercise a materially different capability.

Compare row by row against the table above it. Azure OCR moves from 52.00% to 76.50%; Tesseract from 54.00% to 75.75%; Qwen2.5-VL-7B from 120.00% to 160.75%. Systems tuned for printed Arabic do not transfer to handwriting. More on what that means in practice →

3. Table structure recognition

456 samples · 14 systems. TEDS (Tree-Edit-Distance-based Similarity) scores the recovered table tree against the reference, so it penalises a wrong row span as well as a wrong character.

Against the mean of the thirteen baselines (61.57), Δ = +59.15%. Every one of thirteen table types scored above 93.

Two observations. The ordering by family is clean — purpose-built parsers lead, general closed models follow, open-weight models trail them, OCR-plus-pipeline combinations trail everything — which reverses the image-to-text ordering, where LlamaParse and Landing AI performed poorly at raw transcription. Table structure rewards an explicit geometric model of the page. Second, the spread within the framework family shows the downstream pipeline matters as much as the engine feeding it: the same EasyOCR output scores 49.10 through Docling and 39.09 through Img2Table. Table fidelity is a property of the whole chain.

4. End-to-end PDF-to-Markdown

33 manually annotated multi-page PDFs · 11 systems. The closest proxy in KITAB-Bench for deployment conditions: recognition, reading-order and structural errors all accumulate into one output. MARS is the equally weighted combination of chrF (text fidelity) and TEDS (table fidelity).

5. Layout detection

1,700 pages, the Arabic BCE-Arabic-v1 pool · 6 systems. Class-aware IoU matching with standard object-detection metrics.

Two of the three losses are on this table. DETR in Docling leads on localization (mAP@0.5 0.750 against 0.611, we are 18.53% lower) and on recall (0.725 against 0.711, 1.93% lower). We lead on precision by 46.33% and on F1 by 22.48%.

The shape of the layout result

Precision is high and stable — from 0.750 on footnotes to 0.983 on page footers, with seven of eight categories above 0.86. Recall is the variable component, from 0.428 on captions to 0.949 on page headers. The parser is consistently right about what it detects and inconsistently complete about what it detects — its failures are omissions rather than misclassifications.

Captions are the clear weak point, with a precision-recall gap of 0.473 over 1,230 instances — roughly six captions in ten are not detected as captions. The practical consequence is specific: an undetected caption is not lost text, it is very likely absorbed into the adjacent body-text region, but its association with the figure or table it describes is lost. That degrades figure-aware retrieval specifically.

Scope. Vendor-conducted evaluation on four of the nine KITAB-Bench tasks, 5,949 samples, seventeen baselines. Most baseline figures are quoted from the published KITAB-Bench paper; LlamaParse and Landing AI were measured internally and LlamaParse is the strongest baseline on three of four tasks — see provenance. Image-to-text aggregates are unweighted macro averages over twelve subsets; the sample-weighted CER is 14.08%. Source: Heakl et al., Findings of ACL 2025, arXiv:2502.14949.