The archive
Verification
The risk in work like this is not a visibly garbled word but a line that vanishes without trace. Here is how the result was checked for that.
Two independent checks were run on the finished document: one for dropped content, one for residual OCR error. Both are reproducible — pipeline/5_verify.py runs the first.
1. Coverage — was anything lost?
The risk in a pipeline like this is not a garbled word, which is visible, but a silently dropped line, paragraph or page, which is not. Header-stripping, footnote-splitting, noise-filtering and paragraph-merging each remove or move text, and a bug in any of them could lose a passage without leaving a trace in the output.
So the check works backwards from the source. For every scanned page, four six-word windows were drawn from lines the OCR read confidently, and each was searched for in the finished manuscript. A missing window means text that was on the page is not in the document.
| Pages probed | 264 of 267 (front matter and blank pages excluded) |
| Probes | 1051 |
| Pages with every probe found | 254 |
| Pages with a missing probe | 10 |
All nine remaining pages are accounted for, and in every case the probe fails because the source text was deliberately changed — the check is comparing against the uncorrected OCR reading, which no longer appears:
| Page | Why the probe misses |
|---|---|
| 18 | probe contains dispostions, corrected to dispositions |
| 25 | probe contains Wurttenderg, corrected to Wurttemberg |
| 151 | page re-transcribed by hand; probe contains OCR November l th for November 16th |
| 154 | page re-transcribed by hand; probe contains OCR divices for divides |
| 156 | page re-transcribed by hand; probe contains OCR lest for lost |
| 195 | probe spans the author's *, now a numbered footnote reference |
| 227 | probe contains llth, corrected to 11th |
| 231 | probe contains tne, corrected to the |
| 235 | probe contains nad, corrected to had |
| 244 | probe spans the author's *, now a numbered footnote reference |
No dropped content was found. Two earlier runs of this same check did surface real bugs, which is what it is for: a footnote separator rule that overlapped the first note line (so notes leaked into the body on page 19), and paragraphs on footnoted pages failing to carry across the page break. Both were fixed and the check re-run.
2. Residual error rate
Every word in the finished text was checked against a 160,000-word dictionary, ignoring accents and closed-up compounds.
| Words in document | 67,237 |
| Not in dictionary | 306 (0.46%) |
| Distinct forms | 244 |
Most of that figure is not error. It is the author's own spellings and compound nouns, German and French phrases, and the manuscript's several hundred proper nouns — people, airships, and place names across East Africa, Anatolia and the North Sea. See preserved-spellings.md.
The 20 most frequent unrecognised forms, for reference:
| Form | Times |
|---|---|
von |
23 |
occured |
7 |
airforce |
5 |
rudderman |
5 |
askaris |
4 |
longitudenal |
4 |
anonimity |
3 |
darkeness |
3 |
embarassment |
3 |
machinegun |
3 |
socalled |
3 |
der |
2 |
dilemna |
2 |
disasterous |
2 |
duraluminum |
2 |
elan |
2 |
flieg |
2 |
lle |
2 |
reconaissance |
2 |
vertretbare |
2 |
3. Structural checks
| Check | Result |
|---|---|
| Chapters matched against the table of contents | 15 / 15 |
| Chapter order | matches the table of contents |
| Footnotes carried through | 31 / 31 |
Footnotes anchored at the author's * |
29 / 31 (2 placed at paragraph end, where the asterisk itself was lost to the scan) |
| Paragraphs not ending in terminal punctuation | 8 of 522 — every one ends in a colon introducing a quotation, or in a footnote marker |
| Paragraphs beginning lower-case | 0 |
| Running headers leaked into body text | 0 |
| Dangling line-break hyphens | 0 |
| Word-level XSD validation of the .docx | passed |
The last two counts are worth dwelling on, because they are how three real bugs were caught. A paragraph that stops mid-sentence, or that starts in lower case, almost always means a page-break merge that failed. Chasing the last few surfaced: a noise filter discarding the genuine one-word line "overcast." because it was faint and short; block quotations whose line-break hyphens were never rejoined, because quotes kept their raw lines while body text did not; and a full stop on page 166 that the scanner had read as a comma. None of the three would have been visible just by reading the output.
4. Visual spot-checks
Rendered pages were compared against the corresponding scans by eye: the title page, table of contents, the Prologue opening, a footnoted page, a block-quotation page, the German verse, and Chapter IX (the hand-transcribed section). Crops of the pages that settled a specific question are in evidence/.