Arabic OCR glossary

Character Error Rate (CER)

The normalized edit distance between the recognized text and the reference, counted in characters: the number of insertions, deletions and substitutions needed to turn the output into the reference, divided by the reference length. Lower is better.

When it misleads: CER treats every character as equally important. A system that misreads a digit in an account number and a system that misreads a space score the same. It is also alignment-brittle — a single spurious insertion early in a line can cascade into a much larger score than the perceived error justifies.

The trap: CER is a ratio. A CER of 0.125 and a CER of 12.50% are the same quantity. Writing it as 0.125% overstates accuracy by a factor of 100 — a mistake common enough that it is worth checking whenever a vendor quotes a suspiciously small number.

Word Error Rate (WER)

The same normalized edit distance, counted in words rather than characters. Lower is better.

Why it is usually higher than CER: one wrong character makes a whole word wrong. A system with a 12.50% CER will typically report a WER around twice that. Comparing one system's CER against another's WER is meaningless, and it happens more often than it should.

Why it matters for Arabic specifically: Arabic word boundaries interact with clitics and affixes, so a segmentation difference can produce word errors where the characters are entirely correct.

Why error rates can exceed 100%

Because edit distance is not bounded by the reference length. If a system emits more content than was on the page, the number of deletions needed to recover the reference can exceed the reference itself.

In the KITAB-Bench measurements, one baseline records a 495.00% character error rate on the full image-to-text task and 610.75% on handwriting. These are real measurements, not data errors. A system in that range is not misreading the page — it is generating text that was never there.

Practical reading: an error rate above roughly 100% is a signal of hallucination rather than of poor recognition, and the two failure modes have very different consequences for a document archive.

chrF

A character n-gram F-score between output and reference, on a 0–100 scale. Higher is better.

Why it exists: it does not require a single global alignment, so it is less brittle than edit distance when content has been reordered rather than misread. It is the useful auxiliary measure when CER and WER disagree with each other.

BLEU and METEOR

Machine-translation measures sometimes reported alongside OCR results. Higher is better.

When they mislead badly: BLEU requires four-gram matches. On a corpus of isolated handwritten words there are few or none, so BLEU can return 0.00 for output that is substantially correct. In our own results the ADAB corpus returns a BLEU of 0.00 alongside a chrF of 75.10 and a CER of 12.00% — a limitation of BLEU on short segments, not a recognition failure. Treat BLEU on OCR with suspicion.

TEDS — Tree-Edit-Distance-based Similarity

Compares the recovered table, represented as a tree, against the reference tree. Higher is better, 0–100.

Why it is the right measure for tables: it scores structure as well as content. A system that reads every character correctly but merges two cells, or loses a row span, is penalised — correctly, because a table with the right characters in the wrong cells is worse than useless for anyone querying it.

MARS

An equally weighted combination of chrF (text fidelity) and TEDS (table fidelity), used for end-to-end document conversion. Higher is better.

What to watch: because the two components are weighted equally, a system can lead on MARS while trailing on text. Ours does exactly that on the KITAB-Bench PDF-to-Markdown task — chrF 79.81 against LlamaParse's 82.51, but TEDS 80.12 against their 70.36. Read the components, not just the composite.

Document layout analysis

Identifying and localizing the structural regions of a page — headers, footers, section headings, body text, figures, tables, captions, footnotes — and assigning each a type.

Why it matters more than it sounds: it is the stage that converts a flat transcription into a navigable document. For retrieval-augmented systems it is what makes semantically coherent chunking possible. Without it a chunker splits on character counts and routinely severs a heading from the section it introduces.

mAP, precision and recall in layout

mAP@0.5 counts a predicted region correct when it overlaps the reference by at least 50% intersection-over-union. Precision is the share of detections that were right. Recall is the share of true regions that were found. F1 is their harmonic mean. All higher-is-better.

Why the split matters: a detector can be highly precise and poorly complete, or the reverse, and the two failure modes have opposite consequences. High precision with low recall means what you get is trustworthy but incomplete. Low precision with high recall means you have everything plus noise you must filter. Our own layout result is the first kind — precision 0.916, recall 0.711.

Macro versus micro averaging

A macro average is the unweighted mean across subsets: a 10-sample subset counts as much as an 800-sample subset. A micro or sample-weighted average weights by subset size and estimates expected performance on a randomly drawn page.

They can differ substantially. On the KITAB-Bench image-to-text track our own figures are 12.50% macro-averaged and 14.08% sample-weighted. Both are correct; they answer different questions. Macro asks "how does this system do across the range of material types"; micro asks "what should I expect on a page drawn at random".

What to check in any benchmark: which convention was used, and whether it is the same one used for the baselines. A macro figure compared against a micro figure is not a comparison.

Reading order

The sequence in which regions of a page should be read. A distinct problem from recognition: a system can transcribe every character correctly and still return them in an order that makes the document unusable.

Arabic is right-to-left, and mixed Arabic–Latin content — common in technical, legal and financial documents — produces bidirectional sequences that are a frequent source of error. Multi-column layouts compound it.

Ligatures and connected script

Arabic letters change shape depending on position and join to their neighbours. A letterform has initial, medial, final and isolated variants, and certain pairs form obligatory ligatures.

Consequence for OCR: character segmentation is not a solved preliminary step the way it is for Latin print. Systems that segment before recognising tend to fail on connected script, which is part of why traditional engines degrade so sharply on Arabic handwriting.

Diacritics (tashkīl)

Optional vowel and pronunciation marks written above and below the base letters. Most modern Arabic text omits them; Qur'anic, poetic, pedagogical and some historical material includes them.

Why they complicate measurement: whether diacritics are counted as characters materially changes a character error rate on diacritised material. A benchmark that normalises them away and one that does not are not comparable.

OCR hallucination

Text produced by a recognition system that was never present on the page.

Why it is worse than a misread character: the output is fluent and plausible rather than obviously wrong, so it survives review. A garbled word is visibly garbled; an invented sentence in correct Arabic is not. It is a characteristic failure mode of generative vision-language models pressed into OCR duty, and it is the main reason an error rate above 100% should change how you read a leaderboard rather than just where you place a system on it.