This is the procedure named on our editorial standards page. It explains how we check reference works that are far too big to read entry by entry.
Rather than describe it in the abstract, we will walk through one real job: the Manx dictionary John Kelly compiled before his death in 1809, printed in 1866. Every figure below comes from the files that job produced.
What we started with
A scan held by the Internet Archive, made from Cornell’s copy. We downloaded it on 27 July 2026 and recorded a fingerprint of that exact file, so every later step could be traced back to it. We re-checked the fingerprint on 11 August to confirm nothing had shifted underneath us.
Before using it we confirmed it was out of copyright. Kelly died in 1809 and this printing is 1866, long before any threshold that would still apply, and we read the title page of the scan itself rather than relying on a catalogue entry.
What goes wrong in a scan
A scan is a photograph. To search or reuse it, the words have to become text, and that is done by software reading the picture and guessing at each letter. The process is called optical character recognition, usually shortened to OCR. Our scan came with an OCR attempt already made.
That attempt was built for modern English books. This is a Victorian dictionary of a Celtic language, set tight, with abbreviations in small italic. So it makes mistakes.
The useful part is that it makes the same mistakes over and over. A machine that fails is at least consistent about how it fails.
Take the abbreviation that marks a part of speech. Kelly’s own table lists s. for “Substantive or Noun”, substantive being the older word for a noun. In the scan that small italic s. is often read as a guillemet, the French quotation mark «. It happens 1,435 times.
The guillemet is not scattered randomly. It sits exactly where an abbreviation belongs and nowhere else, and the entries carrying it are all nouns: a relapse, a lodging for a day, a calm at sea.
These are reference books. Somebody is looking a word up in order to learn it. An entry saying the word is a « rather than a noun is not untidy. It teaches something false.
But look again at that number. The machine did not make 1,435 different mistakes. It made the same mistake 1,435 times, and an error you can describe exactly is an error you can correct exactly.
How we repair it
Not by reading 80,216 lines, and not by asking a machine what the page probably said. We write a rule for each repeated error.
A rule is a find-and-replace instruction, written precisely enough that a computer can apply it without judgement. When you see exactly this, put exactly that instead. One error, one correction, nothing else.
Here is the guillemet rule as it runs:
pattern: ^([A-Z][A-Za-z'\-]{1,26},)\s*[«»]\.\s+
replacement: \1 s.
In plain words: at the start of a line, find a capitalised word up to 27 characters, then a comma, then a guillemet and a full stop. Keep the word and the comma exactly as they are, and put s. where the guillemet was.
The rule is narrow by design. A guillemet anywhere else in the book is left alone. It cannot spread, and it cannot decide a word looks wrong and improve it. That is what lets us run it across 1,435 entries nobody has read and still say exactly what happened to them.
Kelly needed 18 such rules, and they changed 7,863 lines out of 80,216. Every rule is recorded with the count it matched and a one-line reason, and every individual change is written down with the text before and the text after.
Rules do not transfer between books. Those written for our Irish sources are calibrated for Gaelic type and were not carried over here. Cregeen’s 1835 dictionary, in the same language and only thirty years later, still needed its own set of 13. Two of our projects reached that conclusion separately, working on different languages, so we now treat it as settled.
Where AI comes in, and where it does not
We use AI to build tools. The tools are what touch a restored book.
A language model predicts words. Asked to repair a damaged page it will produce something plausible, and plausible is exactly the wrong thing when the page is a two-hundred-year-old dictionary and the reader is trying to learn from it.
So we do not point a model at the pages of a restored book. We use one to help write the software: the rules, the parser, the checkers. Those programs are deterministic. Given the same input they produce the same output every time, and they cannot invent a word that was not on the page.
You can read all eighteen rules in an afternoon and know precisely what was done to this dictionary. You could not do that if a model had been asked to tidy it up.
The words in the finished dictionary are John Kelly’s and William Gill’s.
What rules cannot fix
Some damage is not repeatable, and a rule is no use against it. Those lines get flagged and queued for a person rather than quietly patched. Several thousand from this book are waiting.
A restoration that reports no problems is either perfect or not looking.
One thing helps here. Gill credits Cregeen’s earlier dictionary in a few thousand entries, so for those we have a second nineteenth-century source saying the same thing.
What gets reviewed
The rules, and a sample of the result. Not the whole thing.
This one dictionary yielded 27,588 entries. Nobody reads that.
But there are only eighteen rules. Judging whether each one is a sound reading of the language is a job a person can genuinely do in about half an hour, and it is the job that matters. A wrong rule is wrong 1,435 times. A wrong entry is wrong once.
What a review pack looks like
One section per rule, giving the rule itself, how many times it fired, at least two real before-and-after examples with line numbers, and a verdict box.
Rule 2,
pos_s_guillemet, fired 1,435 times.
Before: Air, «. aer. After: Air, s. aer.
Before: Gut, «. tedd. After: Gut, s. tedd.
Verdict: ( ) sound ( ) wrong ( ) unsure
The reviewer is not asked whether those two lines came out right. They are asked whether a guillemet with a full stop, in that position, really means s. in an 1866 Manx dictionary. Someone who knows the language can answer that, and answering it once settles all 1,435 entries.
Alongside the rules we read a random sample of finished entries in full. That half is still outstanding on this book, and the section below says so.
The sample is drawn using a seed, a starting number for the random selection. The same seed always picks the same rows, and we publish ours, so anyone can regenerate the identical sample and check exactly the entries we checked.
| Entries in the batch | Read in full |
|---|---|
| 50 or fewer | all of them |
| 51 to 500 | 30 |
| over 500 | 50 |
On each entry we check that it names a source, that the source actually says what we claim, that copyrighted text is not quoted beyond fair use, that modern revivals are marked apart from historical forms, and that anything sensitive is flagged. Sensitive material is read in full and never left to the sample.
What happens when a check fails
A rule marked wrong gets fixed and the whole work is rebuilt. A rule does not fail once; it fails every time it fired.
A bad entry in the spot-check is different. One error, and we correct it and double the sample. Two or more, and the batch does not publish, or comes down if it is already out, and we do it again. Any sensitivity failure stops the batch on its own, whatever the count.
Does the repair work?
Only where we have something to measure against, which means a second, already-correct copy of the same text. We have one: a clean transcription of the 1904 Dinneen dictionary, held privately as a yardstick.
Two standard measures apply. Character error rate counts how many individual letters are wrong. Word error rate counts how many whole words are wrong, and since a word counts as an error if any letter in it is wrong, it is always the harsher number.
The rules cut the character error rate from 0.297 to 0.274, an improvement of 7.9%. Roughly, about thirty letters in every hundred were wrong before and about twenty-seven after. The word error rate did not move, staying at 0.584.
That gap is the finding. The rules fix letters inside a garbled word, so the page becomes readable, but they rarely make the whole word right. Better legibility, not better accuracy.
We repeated it across twelve passages spanning six letters of the alphabet to see whether the first result was a fluke. It was not: 8.1% on the character rate, every passage improved, the word rate flat again.
Three caveats:
- It is one dictionary. Our own working notes say a figure fit to publish as a headline needs wider coverage than this, so treat it as an early reading rather than a settled one.
- The scanning engine matters far more than our rules do. A better one improved the text roughly four times as much. That comparison comes from a rougher measurement, so take it as an order of magnitude rather than a precise ratio.
- We have this yardstick for one language reference only. For most sources no clean copy exists, and until one does we cannot put a number on the repair at all.
What is not finished
Kelly’s review pack has been built and has not yet been worked through. Until it is, that record fails our own automated check, and it should.
There is no error rate for this book yet either. That needs the corrected sample the pack is waiting for, and the same is true of every language reference where no independent clean copy exists to measure against.
The record for this job also states, in its own opening line, that it was written the day after the work rather than during it, and is therefore reconstructed from surviving files rather than a contemporaneous note.
When the review is done, the outcome goes into the work’s provenance record: who reviewed it, when, which rules passed, the sample seed, and the result. No record means the review did not happen.
What this proves
Not that every entry is right. It proves that a person examined the method, that anyone can repeat that examination from the published seed, and that a stated share of the output was read against the source.
Where we have read everything, we say so. Where we sampled, we say that. Where we have not finished, this page says that too.
If you find something wrong here, please tell us. Corrections from readers have already improved this work.
Responsible: Rosmerta Publishing LLC. Effective 13 August 2026, revised 19 August 2026. Reviewed yearly, and whenever our practice changes.