Catalogue · How we recognise texts

How we recognise texts

Some of the books and documents the decolonial.ist knowledge base relies on existed online only as scans, or with a broken text layer. Such files cannot be searched, and a quotation cannot be checked without leafing through images. We recognise these texts ourselves and publish them next to the scan. Every such text has a block "How the text was obtained" on its material card: what it was recognised with, how it was checked, and what is still wrong with it.

What we recognise with

Tesseract is free recognition software. It reads words correctly, but on pre-reform Russian typography it garbles the letters ѣ, і, ъ, ѳ, ѵ.

Qwen 3.6 (35B) is a language model that sees the page image. We do not let it read a page from scratch: given a whole page, it substitutes words ("I will not" became "I will"). The model only corrects letters in the Tesseract text or in the publisher's text layer. The program checks each of its corrections: an edit that changes only look-alike or pre-reform letters is accepted; an edit that changes the word itself is rejected, and the Tesseract word stays. The model looks at rejected edits a second time, one word at a time, cut out of the scan.

whisper large-v3-turbo is a speech recognition model. It produced the transcripts of audio from video recordings and broadcasts. These are unproofread drafts: where speech is drowned by music or noise, they have gaps and repetitions.

How we check

Before a run, the method is checked word by word against the scan on sample pages. During the run, the program checks every edit by the model, as described above. Quotations in the knowledge base are checked against our text by a comparison script, and against the scan by hand where the text raises doubt. The check figures for each volume are on its card.

Where we go wrong

Tables, small-print footnotes and marginal notes may be out of order. Where Tesseract erred in a word rather than a letter, the error remains. Chechen alphabets of the 19th and 20th centuries (Uslar 1862, Bartholomaei 1866, Serdalo 1938) the model does not read but invents, so those editions are published as scans only. Audio transcripts are not proofread.

What we are improving

A sample check of every recognised volume against the scan, with an error count per thousand words - that figure will go on the card. E. N. Kusheva's books: the 1957 volume of documents is being re-recognised in full. Chechen alphabets: recognition of the Russian parts, with letter tables done by hand. Audio transcripts are proofread as they are quoted.

Found an error in a recognised text? Write to contact@decolonial.ist.