19,409 Greek letters, and other things we learned testing OCR

Our OCR wrote the Greek letter α in place of a plain "a" 19,409 times. On screen the two look almost the same. A search for the word finds nothing.

We found it while testing how Plainlamp should read scanned PDFs. Plainlamp turns a hard PDF, such as a paper, a policy or a standard, into a short lesson video and a study guide. Every point in the lesson names the page it comes from. That only works if we can read the page.

Why scans are hard

Most new PDFs carry their text inside the file. Older ones, and anything printed and scanned, do not: to a computer, each page is a picture. OCR (optical character recognition) turns the picture back into words. It also has to put the words in the right order, and on a page with two columns, footnotes and headers, that is the hard part.

Our first OCR setup read the words well. On two-column pages it put lines in the wrong order, so we wrote rules about columns. Those rules were tuned on one journal's layout. So we built a proper test.

The test

We used 32 published articles from 1996 to 2026, mostly from management journals. 17 still have their text inside the PDF, so we know their true text. 14 are real scans. One has a single scanned page.

For the 17 with known text, we made a picture of each page that looks like a photocopy, ran OCR on it, and compared the result with the true text. For the real scans there is no true text. There a blind AI reviewer checked the reading order: a separate AI model sees the page picture and two readings, without knowing which setup made which.

We call each setup of OCR tools a "route":

Route 1 is what we had: RapidOCR 3.9.2 running PaddleOCR's PP-OCRv6 model, with Tesseract as a second reading, and our own column rules for the order.
Routes 2 to 5 change one part at a time: a faster letter model (PP-OCRv6 tiny), a letter model trained on Latin-script languages (PP-OCRv5 Latin), and a layout model (PP-DocLayoutV3). The layout model finds the columns, headings, footnotes and page numbers, and gives its own reading order.
Route 6 uses PP-OCRv5 Latin and PP-DocLayoutV3 together. It is the best free route.
Route 7 is a paid cloud service, Mistral OCR 4.1.

We tried two more services. Amazon Textract is not offered in the AWS region we work in. Amazon Nova Lite is, and it read 19% of the words on our 34 test pages. We also had Claude Haiku 5.5 read the page pictures. It read about as well as RapidOCR on the pages it finished, but the content filter stopped it after 18 of 34 pages. We think that is because a transcript of a published page is a word-for-word copy of it. It also gives no position on the page, so it cannot be our OCR.

What we found

The Greek α. On the 16 articles every route read, the true text has no α at all. Route 1 wrote it 5,332 times, route 6 555 times, Mistral not once. The fix was small: after OCR, Plainlamp now turns an α inside an English word back into "a".

Chart. The word "practice" twice: with a Latin a, which a search finds, and with a Greek α, which a search misses; on screen they look almost the same. Times α was written on 16 articles whose real text has none: route 1 (PP-OCRv6 small) 5,332, route 6 (PP-OCRv5 Latin) 555, Mistral OCR 4.1 none. On all 31 articles, route 1 wrote α 19,409 times.

Order matters more than letters. Here are all the routes on the photocopies:

Route	Characters right	Lines found	Lines in order	Engines agree (real scans)	Seconds a page
1, PP-OCRv6 + column rules	87.3%	96.5%	95.4%	85.5%	13.7
2, + layout model	90.9%	96.5%	96.5%	85.5%	about 16
3, PP-OCRv6 tiny	87.4%	96.6%	95.3%	85.9%	9.2
4, PP-OCRv5 Latin	87.3%	97.9%	95.3%	90.0%	17.1
5, tiny + layout model	90.9%	96.6%	96.5%	85.9%	about 11
6, Latin + layout model	91.0%	97.9%	96.5%	90.0%	about 19
7, Mistral OCR 4.1	92.1%	98.6%	96.5%	(one engine)	about 1

"Characters right" compares the whole page, in order, with the true text, so a line in the wrong place counts as errors. That is why the layout model moves it from 87.3% to 90.9% while the Latin letter model alone leaves it at 87.3%. Looking article by article, route 1 actually read more characters right than route 6 in 12 of the 16 articles. Its average falls because of one article where its order broke. The Latin model's real gain shows on the real scans: the two engines agree on 90.0% of lines instead of 85.5%.

Chart: lines put back in the right order on fake photocopies, as an average per article on 16 articles. Route 1 (RapidOCR with PP-OCRv6 and Tesseract, what we had) 95.4%, route 6 (PP-OCRv5 Latin and the PP-DocLayoutV3 layout model) 96.5%, Mistral OCR 4.1 96.5%. All three are within about one point.

The layout model fixed most of the order, and may lose some text. We took the 37 real-scan pages where our column rules and the layout model ordered the lines differently, both with the same letter model. The reviewer found only the layout model in order on 23 pages, only the column rules on none, both on 9 and neither on 5. The column rules usually failed the same way: a footnote at the foot of the left column landed in the middle of a sentence that runs on into the right column. But the reviewer reported missing text 18 times for the layout model and 7 times for the column rules. On two pages it ran table cells together.

Chart: a layout model fixed most of the reading order. On 37 real-scan pages where our column rules and the PP-DocLayoutV3 layout model gave a different order, the blind AI reviewer found only the layout model in order on 23 pages, both on 9, neither on 5 and only our column rules on none. But it found text missing 18 times for the layout model and 7 times for our column rules.

Free or paid. Route 6 and Mistral read 118 real-scan pages differently, and each failed in its own way. We counted the reviewer's findings by kind, because its verdict on a page fails it for missing text as well as for order. Mistral put a footnote inside a sentence on 5 pages; route 6 put a footnote or a stray line inside one on 2. Leaving out the 5 pages with a figure, route 6 scrambled a table, reference list or notice on 18 pages, Mistral on 2. Route 6 left text out 21 times, 11 of them in one article's tables; Mistral left none out (the reviewer's two entries for it are misread words). But Mistral left out the labels of 4 of the 5 figures, which the reviewer's rules do not count, and route 6 kept them all, out of order. Misread words are not in these counts, and route 6 has far more of them. Mistral is not better everywhere: on our longest article, 64 pages, it got 73.8% of characters right against 83.7% for route 6.

Chart: free or paid, each failed in its own way. On 118 real-scan pages that route 6 (free) and Mistral OCR 4.1 (paid) read differently, checked by a blind AI reviewer: pages with a footnote or stray line inside a sentence, route 6 2, Mistral 5; pages with a scrambled table, list or notice, route 6 18, Mistral 2; times the reviewer found text left out, route 6 21, Mistral none. Not in the bars: misread words, far more often in route 6 (such as Greek α), and the 5 figures: Mistral left out the labels of 4, and route 6 kept all 5, out of order.

Speed, price and rough edges

On one 8-core CPU, route 1 took 13.7 seconds a page and route 6 about 19. That is the whole job: making the page picture, two readings and the layout. Mistral answered in about 1 second a page. It read 876 pages for about US$3.40, so about US$3.90 per 1,000 pages. The free routes cost only computer time. In Cloudflare Containers, route 6 read the same 17 pages in 127 seconds with the document split over 4 containers, for about US$0.62 per 1,000 pages.

Chart: time per page. Route 6 on our computer about 19 seconds, route 6 in Cloudflare with 4 containers at once 7.5 seconds, Mistral OCR 4.1 about 1 second. Price per 1,000 pages: route 6 in Cloudflare about US$0.62 (measured), Mistral OCR 4.1 about US$3.90.

Both have rough edges. Sent in batches of 8 pages, Mistral returned 9 pages empty with no error; each read fine when sent alone. It returns paragraphs, not lines, and it reads some tables and figures as pictures. Our free routes failed when we ran three documents at once on one computer. And route 6 failed one article even alone: it sent pages in fours and gave up waiting before slow pages were done. We now send one page at a time; we have not run that article again yet.

A cloud service also means the document goes to one more company. For a policy someone brings from work, that counts.

How sure we can be

Every route read the same pages. So to compare two routes, we count the pages that only one of them got right, and for the photocopies the articles on which one scored higher. If the two were equally good, those would split about evenly between them. We call a gap too big for chance when chance alone gives a split at least as uneven less than 1 time in 20 (an exact sign test).

Real scans, order: the layout model was in order on 23 pages where our column rules were not, and never the other way: too big for chance (less than 1 time in a million).
Real scans, missing text: the layout model lost text on 3 pages where the column rules lost none, and never the other way: could be chance (about 1 time in 4). The reviewer's 18 reports of missing text for the layout model, against 7, look like more, but 10 of the 18 are on one article of 3 pages.
Route 6 and Mistral, by kind of fault: the reviewer's verdict on a page fails it for missing text as well as for order, so here we count its findings by kind. A footnote or stray line sat inside a sentence on 2 pages for route 6 where Mistral had none, and on 5 for Mistral where route 6 had none: could be chance (about 1 time in 2). Leaving out the 5 pages with a figure, route 6 scrambled a table, reference list or notice on 17 pages where Mistral did not, and Mistral on 1 where route 6 did not: too big for chance (about 1 time in 6,900).
Route 6 and Mistral, text left out: route 6 left text out on 12 pages where Mistral left none, and never the other way: too big for chance (about 1 time in 2,000). This does not count misread words, which the reviewer was not asked to look for (on these pages, route 6's lines hold the Greek α 281 times, Mistral's never), or figure labels, which its rules do not count: Mistral left out the labels of four of the five figures, and route 6 kept them all, out of order.
Photocopies, lines in order: route 1 put more lines in order than route 6 on 13 of the 16 articles, by at most 2.8 points: too big for chance (about 1 time in 47). Route 6's average is higher only because of one article where route 1's order broke: 63.1% of lines in order against 93.8%. Mistral was ahead of route 6 on 11 articles and behind on 5, which could be chance (about 1 time in 5). So on clean photocopies our old column rules may keep the order a little better on most articles, but they broke badly on one; on real scans the layout model was in order far more often (above), and that is what our choice rested on.
Photocopies, characters right: route 1 was ahead of route 6 on 12 of the 16 articles, which could be chance (about 1 time in 13).
Articles, not pages: the real-scan pages come from 12 and 14 articles, and a route's faults may bunch in one article. Counted article by article (an article counts for the route with more pages right in it), route 6's scrambled tables and lists could be chance (7 articles to 1, about 1 time in 14), route 6's text left out against Mistral still holds (6 articles to 0, about 1 time in 32) and the layout model's lead on order still holds (10 articles to 0, about 1 time in 510).
One reviewer, one read: these counts take the reviewer's verdicts as fixed. It judged some footnote pages differently from one comparison to the next (see the limits), and these tests do not count that.
Many comparisons: this section makes 8 comparisons, and among that many, one gap can pass the 1-in-20 line by chance alone. Allowing for that (Holm's method), route 1's lead on photocopy lines in order could be chance too; the other gaps called too big for chance still hold.

Limits

The photocopy scores come from 16 articles, mostly from one field. Our photocopies are made by us; real ones may be worse.
The scores are averages per article, not over all pages.
One AI reviewer read each page once per comparison, and it judged some footnote pages differently from one pack to the next.
We chose to count route 6's and Mistral's faults by kind (leaving out the figure pages and the misread words) after we had seen the reviewer's findings, to correct a first reading of them. The chance lines do not allow for that.
Speed was measured on one computer and one article, and the Cloudflare run on one article.

What we chose

We chose route 6, with Tesseract kept as the second reading. It runs in Cloudflare Containers in the EU, and each document is split over 4 containers that read at the same time. On our 17-page test scan that took 127 seconds and cost about one US cent, so about US$0.62 per 1,000 pages. Mistral reads more of each page and is faster, but it costs about six times as much, and every document would go to one more company. Route 6 loses more text, so that is what we watch: every OCR page in a Plainlamp lesson is marked for review, and every quote is checked against the page by a second reading. The α fix and one page per OCR request are already built in.