# The China Affair — transcription archive

The working record behind the digitisation of Ernest Walton's *The China Affair*,
a 267-page typewritten manuscript from 1982, scanned as images with no text
layer. This archive holds the finished transcription, the code that produced it,
and the evidence for every judgment made along the way.

The manuscript is a history of the German naval airship **"L 59"** and the 1917
"China Operation" — an attempt to fly medical supplies and ammunition from
Bulgaria to General von Lettow-Vorbeck's forces in German East Africa. The
prologue opens with Commander Looff taking afternoon tea with his British captors
in the Lukuledi valley; the epilogue closes on the rigid airship's disappearance
from history. There is a pleasing detail buried in Chapter I: Count Zeppelin
sailed home from New York in 1863 aboard a steamship named *China*.

## What's here

```
manuscript/    the finished transcription — .docx, .txt, .md
pipeline/      the scripts, numbered in the order they run
data/          the two hand-built inputs, plus machine-generated intermediates
reports/       corrections, preserved spellings, verification, statistics
evidence/      the page images that settled a specific question
```

### Start here

- **`reports/corrections.md`** — every change made to the raw OCR, with the scan
  page it was applied on, grouped by kind of error.
- **`reports/preserved-spellings.md`** — the more interesting half of the job:
  what was deliberately *not* changed, and how the author's own typing errors
  were told apart from the scanner's.
- **`reports/verification.md`** — how the result was checked for silently dropped
  content, and the real bugs that check caught.
- **`reports/statistics.md`** — source, OCR, structure and output figures.

## Page numbering

Page numbers everywhere in this archive are **scan page numbers** — position in
the PDF. The typist's own folio numbers, hand-written at the foot of each page,
run three lower: scan page 230 carries the folio "227".

## How the pipeline works

The scan has no text layer, so everything is reconstructed from pixels. The
central problem is not reading the characters — Tesseract does that at about 95%
confidence on this typescript — but recovering the *structure*: where paragraphs
begin, which text is a footnote, which is a quotation, and which marks on the page
are not text at all.

Almost all of that comes from geometry. Tesseract's TSV output gives a bounding
box and a confidence score for every word, and the typewriter's own habits do the
rest:

| Signal | What it identifies |
|---|---|
| Line left edge ≈ page margin | body text continuing |
| Left edge ≈ margin + 150–280 px | first line of a new paragraph |
| Left edge ≈ margin + 190 px or more, block narrower than the body measure | a quotation |
| Line gap ≈ 100 px | double-spaced body |
| Line gap ≈ 50 px | single-spaced footnote or quotation |
| A solid horizontal run of dark pixels low on the page | the footnote separator rule |
| Centred, short, near the top of a page | a chapter heading |

Two of those deserve a note, because the obvious approach fails. **Indent depth
is not constant** — it ranges from about 150 px to 276 px depending on which
revision typed the page — so a fixed threshold cannot separate a paragraph
opening from a quotation. What does separate them is width: quoted blocks are
narrower than the body measure. And **line spacing does not identify footnotes**,
because block quotations and verse are also single-spaced. The reliable marker is
the separator rule, which is found by scanning the pixels directly rather than by
asking the OCR about it.

One linguistic rule matters as much as any of the geometric ones: a new paragraph
can only follow a completed sentence. On faint pages the OCR sometimes captures
only the right-hand part of a line, so its left edge looks like an indent; that
test rejects the false paragraph breaks this produces.

### Running it

Stages 1–3 need the rendered page images and the OCR output, which are not in
this archive (2.1 GB and 3 MB respectively). To rebuild from the original PDF:

```bash
# render and OCR — about 5 minutes on 4 cores
pdftoppm -gray -r 300 "The China Affair manuscript (scanned 7-26).pdf" /tmp/img/p
for f in /tmp/img/*.pgm; do
  OMP_THREAD_LIMIT=1 tesseract "$f" "/tmp/ocr/$(basename "$f" .pgm)" \
    --psm 3 -l eng tsv
done
```

`OMP_THREAD_LIMIT=1` matters: Tesseract is internally multi-threaded, and running
four workers without it causes enough contention to slow the job roughly
twenty-fold.

```bash
python3 pipeline/1_parse_lines.py        # TSV  → line geometry
python3 pipeline/2_find_rules.py         # find footnote separator rules
python3 pipeline/3_clean.py              # → data/structure.json
node     pipeline/4_build_docx.js        # → manuscript/*.docx
python3 pipeline/4b_export_text.py       # → manuscript/*.txt, *.md
python3 pipeline/5_verify.py             # coverage check
python3 pipeline/6_make_reports.py       # → reports/corrections.md, statistics.md
python3 pipeline/6b_make_verification.py # → reports/verification.md
python3 pipeline/6c_make_evidence.py     # → evidence/
```

Stages 5 and 6b read a plain-text rendering of the built document at
`/tmp/final.txt`; produce it with
`pandoc -t plain --wrap=none "manuscript/The China Affair - Ernest Walton.docx" -o /tmp/final.txt`.

Stage 3 reads `data/overrides.json` and `data/fixes.json`. Those two files are the
human input to the whole process and the ones worth keeping — everything else
regenerates. The archived scripts resolve their paths relative to the archive
root, and the sequence above was run from this directory to confirm it works:
every part of the rebuilt `.docx` is byte-identical to the delivered one except
the embedded creation timestamp.

Requires `tesseract`, `poppler-utils`, `pandoc`, Python with `pillow`, `numpy`
and `pyspellchecker`, and Node with the `docx` package.

### The two tools

`pipeline/tools/` holds the review harnesses rather than pipeline stages.
`suspect_lines.py` and `review_residue.py` find words the finished text still
cannot account for, then crop the source line for each and tile them into contact
sheets. Reading a sheet of twenty suspect lines at once is what made it practical
to check the long tail by eye instead of guessing — and it is what turned up
`sletiakéel BeleGtiin` as "elevator helmsman", and `euarLéudily` as "curiously".

## Data files

| File | |
|---|---|
| `overrides.json` | Hand transcriptions of scan pages 150–156, the faint carbon copies. Human input. |
| `fixes.json` | Verified OCR corrections, as phrase replacements and token replacements. Human input, each entry checked against the page. |
| `structure.json` | The reconstructed manuscript: an ordered list of typed blocks (paragraph, heading, quote, verse, footnote) with source page numbers, paragraph ids and footnote anchors. This is the useful intermediate — anything else could be rendered from it. |
| `footnote-rules.json` | Detected separator-rule position per page. |
| `corrections-log.json` | Machine-readable version of the corrections log, with page references. |

## Known limitations

- About **0.46%** of words are not in a standard English dictionary. Much of that
  is the author's own spelling, German and French, and several hundred proper
  nouns; the remainder is a thin tail of single-occurrence OCR slips on faint
  lines, left alone rather than guessed at.
- The **Appendix** ("How A Zeppelin Flies") is listed in the table of contents but
  was not part of the scanned manuscript. It is noted as not included.
- **Two of 31 footnotes** are anchored at the end of their paragraph rather than
  at the exact asterisk, because the asterisk itself was lost to the scan.
- The .docx **does not preserve the original pagination**. Text is reflowed into
  standard manuscript format, so the scan's page breaks are gone; `structure.json`
  retains the source page numbers for every block if you need to map back.
- **One word was chosen rather than read** — see section 7 of
  `reports/corrections.md`.
