Adding a language

Nothing about the number eleven is special. French, German, Turkish, Hindi, Spanish and Chinese each arrived as one run of one command — a row of the table and the files that go with it — because every part of Parseh reads the table rather than keeping a list of languages of its own.

1. What a language is#

A language is one row of the registry plus the files that go with it. The registry is two files read as one: lib/languages.json, Parseh’s own languages, which comes with Parseh and is replaced by each update, and config/languages.json, the languages added on this machine, read after it. Where both have a row for the same code, Parseh’s own is the one used. Beside the row:

• lib/lang/<code>.tex — what a reading edition’s LaTeX needs to know about it;
• docs/lang/<code>.md — its annotation conventions;
• three content folders, books/<folder>/, youtube/videos/<folder>/ and markdown/library/<folder>/, each with a .gitkeep so an empty one survives a clone.

The fields of the row that carry a decision:

2. The one command#

lib/newlang.py writes all of it. Run from Parseh’s folder:

python3 lib/newlang.py                 # what it needs, and the languages there arepython3 lib/newlang.py --check         # every language validated, every file presentpython3 lib/newlang.py --help          # every flagpython3 lib/newlang.py ko --name Korean --native 한국어 --script other \    --chars '\uAC00-\uD7AF\u1100-\u11FF' --font 'Noto Serif KR' --iso3 kor

It asks for nothing it can work out — the folder, the tag, babel’s name, the levels, their names and their titles, the labels, and a free pair of Anki ids, checked against every id in both files and every retired one — and works out nothing you ought to decide. The flags that carry a decision:

It refuses, and writes nothing, for a code that is taken or is not two or three lower-case letters, a folder or a tag another language has, a chars that does not compile in both Python and JavaScript, --script other without --chars, --reading on a script that cannot be recognised, --words on a language written with spaces, --digits that are not exactly ten, and verb labels carrying a character LaTeX would read as syntax. A font the machine does not have is only a warning: the row may be written on one machine for another.

What it said for Korean:

adding Korean (ko, 한국어) to Parsehwarnings (none of them stopped anything):  - --font 'Noto Serif KR' is not on this machine.  Kept: the entry may be for another one, and \FrankPickFont falls through tex.main_fallbacks.  config/languages.json entry added -- a language of this machine's, which an update leaves where it is:      folder korean/   tag korean   dir ltr   script other   digits Latin      passes: Sentence (the sentence, to read on its own); Chunks (the sentence cut into chunks, each with its gloss beside it)              (each button wears its name; a person renames a level on their own pages, and this is the default)      fonts:  main Noto Serif KR, alt none, bundled none      anki:   1724645326311 / 1724645326312 (slot 8212631), field Korean  lib/lang/ko.tex      written  docs/lang/ko.md      written  books/korean/        made, with .gitkeep  youtube/videos/korean/ made, with .gitkeep  markdown/library/korean/ made, with .gitkeep

3. What is left to write by hand#

The command fills in the shape and not the prose, and it ends by listing what is yours:

1. \FrankHowTo in lib/lang/<code>.tex — the “How to read this” page at the front of every printed book: what each level is for, and what the marks of the transliteration mean. It is written with a placeholder, so a book builds meanwhile; lib/lang/fa.tex is the voice to follow. For a language with a reading, words or a vertical pass, the file also has TODO blocks — the ruby over a word, the reading pass, the rotated columns — to copy from lib/lang/ja.tex and zh.tex; copy them rather than writing your own, since both are tuned against real pages.
2. docs/lang/<code>.md — the conventions, in seven sections (below).
3. A fixture, if the test suite is to cover the language: a small book under tests/fixtures/books/<folder>/, a video under tests/fixtures/videos/<folder>/, a card in tests/fixtures/anki/. The smoke test only tries a language it has a fixture for.
4. A starter document, markdown/exlex/starters/<code>.md — the page the studio’s + New opens on for this language, showing the marks it really uses. Until it exists the studio builds a plain one from the row.
5. A verb recipe, lib/verbs/<code>.py, which reads a dictionary entry and returns the language’s three verb forms, with an optional table of exceptions in lib/lang/<code>.verbs.json. Without it nothing breaks: the editor’s sources offer every verb as a plain word (\dw) instead of a whole \vb. Try one with python3 lib/verbs/__init__.py <code> "<a sentence>" and pin it with cases in tests/fixtures/verbs/<code>.json.

A language with marks to strip is also reminded that vb_video_bare has to be set by hand if its videos are written without them. Other files a language may have, read only where they exist: lib/lang/<code>.lookup.json (the dictionary’s rules for what is written onto a word), <code>.ipa.json, <code>.chunk.json, and for a worded language <code>.words.json.

The conventions file#

docs/lang/<code>.md starts from docs/lang/_template.md with the language’s name, labels and verb forms filled in. It is instructions to an annotator — and the annotator is usually a model: the video prompt, the new-book prompt and the prompt that has an LLM gloss a stretch of a book or a video (gloss with an LLM) take it in when that language is chosen, each the sections it needs — a stretch, whose chunks are already cut, has no use for Chunking, and the studio’s prompts take only the transliteration, the reading and the note on writing the script. What only a book is told, or only a video, is marked in the template, so that each prompt reads clean for its own. Keep all seven sections, even one that only says the language has no such thing:

1. The text field — what fa must reproduce, verbatim, and what it must never silently correct. The fidelity checks are measured against it.
2. Reading — the reading rule, or one line saying there is none.
3. Transliteration — the scheme, letter by letter, and what is hyphenated: an annotator with only this file must write the line another one would.
4. Vocabulary — what a gloss carries, and the three forms of a verb in the order \vb prints them.
5. Never gloss — the function words that never get an entry, as a list.
6. Chunking — how big a chunk is, and what must never be split.
7. Example — a chunk or two of the language, each with its transliteration, its vocabulary line and a meaning that is aligned: it says what that chunk’s own words say, in the order of the text, and read in a row may not be good prose (What the meaning says). The prompts that ask a model for glosses carry it, so that a model sees the shape of an answer in your language, and the add page’s prompt uses it in place of a video from the shelf. Have a speaker of the language read it.

\FrankHowTo says the same things to the reader; the two must agree. The nearest existing file is the one to model yours on: docs/lang/it.md for a Latin-script language that spells itself, fr.md where the spelling hides the sound, fa.md or ar.md for an Arabic script, ja.md for a reading, zh.md for words without a reading.

4. Giving it a font#

If the device can be trusted to have the face, name it with --font and its fallbacks, and nothing else is needed: a Latin-script language needs no face at all (it is set in the toolbox’s roman), and the CJK languages name the system’s faces rather than carrying tens of megabytes.

If it cannot, the face travels with Parseh:

1. put the .woff2 in lib/fonts/ for the pages, and the .ttf or .otf beside it for LaTeX;
2. name the .woff2 in the row’s fonts.web_files — --web-font does that, and warns if the file is not in lib/fonts/ yet;
3. declare it once with @font-face in lib/parseh.css and once in the studio’s markdown/app/static/app.css, beside the four already there;
4. check its licence allows it: the four bundled faces are under the SIL Open Font License;
5. record the licence, so that the font travels with it: a row in lib/fonts/README.md, the font’s copyright line in lib/fonts/OFL.txt (or a licence file of its own beside the font, if it is not under the OFL), and an entry in FONTS in lib/notices.py, which draws the Licences page. tests/test_licences.py fails until all three name it (Licences and credits).

The CSS stacks themselves need no editing: the per-language stylesheet every page links is generated from the registry when Parseh starts.

5. Checking it#

python3 lib/newlang.py --check       # every language: files, folders, fonts, ids, rangespython3 tests/smoke.py               # the regression run, every rendererpython3 tests/smoke.py --pdf         # ... and the LaTeX builds (minutes)

--check walks the whole registry — both files, marking the languages added on this machine — and prints a line for everything it looked at: the .tex and the .md, that the verb labels and the passes in the registry and the .tex agree, the three folders, that babel has the locale, the fonts, the bundled files, that the Anki ids are the language’s alone, that chars compiles. A MISSING line is a fault — a missing file, a shared Anki id, a range that does not compile, labels that disagree — and makes it exit with a failure. A note is not: a folder that does not exist yet, a font this machine lacks, a locale this TeX installation has not got. --strict counts the notes as faults too.

  ko  Korean (korean)  -- added on this machine      ok       lib/lang/ko.tex (the reading editions' preamble)      ok       docs/lang/ko.md (the annotation conventions)      ok       vb labels 'pres.' / 'past': the registry and the .tex agree      ok       vb forms: dictionary form / second form / third form      ok       passes vocal/chunks: the registry and the .tex agree      ok       books/korean/, youtube/videos/korean/, library/korean/      ok       babel imports 'ko', which is installed      note     tex.main 'Noto Serif KR' is not on this machine and no fallback is either      ok       Anki ids 1724587311241 / 1724587311242 are this language's alone      ok       chars: compiles in Python and JS12 languages: 0 faults, 1 note.

6. After that#

Restart Parseh, and the language is everywhere a language is chosen: the Language selects of Add a book and Add a video, the studio’s LLM prompt page, a new exercise deck, the gloss selects. Its chip appears on a page as soon as that page has something in it. The reading-help page has a row for it in each section: its dictionary is fetched from kaikki.org by its English name, its corpus from Tatoeba by its iso3, and a translation model exists only if Mozilla offers one between it and English. (The character components are for Japanese and Chinese alone.)

The quickest proof that it works end to end is a video: Add a video, choose the language, and the prompt it copies carries your conventions file. The real test is a book, the one door that goes through LaTeX — build its first chapter early, because lib/lang/<code>.tex is proved by nothing until LaTeX has read it.

7. Three examples#

A Latin-script language is the easy case — six of the eleven are:

python3 lib/newlang.py sv --name Swedish --native svenska --script latin \    --iso3 swe --hyphen swedish

Two levels, Sentence and Chunks; an optional pronunciation line; no font, no ranges, no marks to strip. What is left is the prose: \FrankHowTo and the seven sections.

A right-to-left language with its own digits and a face to carry:

python3 lib/newlang.py ur --name Urdu --native اردو --script arabic \    --font "Noto Nastaliq Urdu" --web-font NotoNastaliqUrdu.woff2 \    --alt-font "Noto Naskh Arabic" --alt-key naskh \    --digits "۰۱۲۳۴۵۶۷۸۹" --iso3 urd

--script arabic gives right to left, the Arabic ranges and the harakat to strip, so the row has a bare pass; --alt-font adds a fourth pass in the second face; --digits puts the Persian figures in place of the Arabic ones. Nastaliq is already in lib/fonts/, because Persian uses it — a face that is not needs the four steps above.

A script of its own: the Korean example above — --script other and the Hangul ranges. And a language written without spaces adds --words (each chunk then carries its words and their readings, and gets a reading-alone pass) and, if it can be set in columns, --vertical; the scaffold then leaves TODO blocks in the .tex to copy from ja.tex and zh.tex.

docs/languages.md is the design underneath all of this.