Add the dof-signatures local corpus: DOF signature blocks cut to names

AGENTS.md

@@ -256,7 +256,7 @@ The 2.0 rewrite landed as underscore-private modules beside the v1 code, which 2
 
 **Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `AFTER_GIVEN_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `nameparser/_reading_index.py`'s `BEHAVIORAL` table, a branch in its `compile_readings` adding the words whose reading carries it and a key in its `Compiled` TypedDict, a constant of the same name in `tools/vocab/build.py`'s `CONSTANTS` roster (an import-time assert there holds the two to one list), and the `NAME = _compiled.NAME` assignment in the config module. A property the compiler reads is kept in the reading index by `indexed()`, which reads `BEHAVIORAL`, so a new one reaches a choice of languages too. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A marker that records a clash with a NAME is the other shape: derived from the word's name reading beside the clashing one (`SUFFIX_ACRONYMS_AMBIGUOUS`, `SUFFIX_WORDS_AMBIGUOUS`, and the complement `NON_GIVEN_NAME_PARTICLES` works from), not written as a property, so it too holds by construction (decisions/vocabulary-model, 2026-10-25). A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). A caller's `Lexicon.add()` to a sub-set does **not** add the word to its base, so a caller adding a brand-new word adds it to both. Two traps a gating set met (#752 phase 2i). Where a caller listing a word in the BASE set has always meant "read it as this everywhere" (the 1.4 recipe `c.titles.add("dean")`), the gated words compile OUT of the base rather than into both, or the gate overrides the caller's listing: `TITLES_AMBIGUOUS` is not a subset of `TITLES` for that reason, and a word in both reads as a title. And `_VOCAB_FIELDS` is read whole by two rosters in `_pipeline/_vocab.py`, the capitals shape's "unlisted" (`in_any_wordlist`) and the comma contrast's "claimed as non-name" (`_NON_NAME_FIELDS`), so a new field joins both by default; a field whose words may be names leaves both, which the grid found as `John Smith, DEAN` reversed where C6 had read a credential; #789's `given_name_abbreviations` met it again (`John Smith, JAS` and `Jas SMITH, XYZ` reversed), found by review because its grid held only the dotted spellings the set was for -- a new field's grid holds the spellings it must NOT move too.
 
-**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials --base <base checkout>`, which reads the base from a baseline kept for its package's files and never the shared snapshot beside the corpus (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating; the shared snapshot beside the corpus is one file per source for every worktree, and #752's phase 2f found ctgov reporting a sibling branch's 1,334 joined-credential readings as its own moves against it); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to English titles, honorifics or post-nominals also reads `snapshot.py --local govuk-people` the same way: GOV.UK's display names of ministers, officials and officers, the title run and the post-nominals as an office writes them (`The Rt Hon`, `Sir`, `Baroness`, ranks; `KC`, `CBE`, `KCB`), in 2,093 of whose 6,021 `parse()` read one or both on 2026-10-30. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific titles such as `Guru` in front, `Shekh`, a title there until #752's phase 2j made it one only inside a title phrase, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals --base <base checkout>` the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality; and `local_agreement.py orcid-researchers --base <base checkout>`, ORCID holders' own two boxes (given names, family name) from every country, scored by the country they work or live in -- a population each person split for themselves, academic but worldwide, the one to read for a change a language pack makes; and `local_agreement.py patentsview-inventors --base <base checkout>`, US patents' inventors in the two boxes their filings give, split by a professional filer, with how many people filed under each name in each country, the one to read for how common a name's word is among a country's US-patent inventors. A change to titles or post-nominals, or to the `cs` pack, reads `local_agreement.py cz-candidates --base <base checkout>`, and with `--locale cs` too: Czech election candidates' four boxes (the title before the name, given names, surname, the title after it), scored by election, the only population whose titles are stated apart from the name, 36% of its 170,181 records titled (2026-10-30), nearly all academic degrees; its names are composed from the boxes (`Ing. Jan Novák, Ph.D.`), so it says nothing about how a writer spaces or punctuates them. These are the only real fielded populations outside the committed fields files, and no CI run sees them either.
+**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials --base <base checkout>`, which reads the base from a baseline kept for its package's files and never the shared snapshot beside the corpus (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating; the shared snapshot beside the corpus is one file per source for every worktree, and #752's phase 2f found ctgov reporting a sibling branch's 1,334 joined-credential readings as its own moves against it); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to English titles, honorifics or post-nominals also reads `snapshot.py --local govuk-people` the same way: GOV.UK's display names of ministers, officials and officers, the title run and the post-nominals as an office writes them (`The Rt Hon`, `Sir`, `Baroness`, ranks; `KC`, `CBE`, `KCB`), in 2,093 of whose 6,021 `parse()` read one or both on 2026-10-30. A change to Spanish titles, to the `es` pack, or to how a title marks where a name starts after an office reads `snapshot.py --local dof-signatures` the same way: the names in the signature blocks of Mexico's Diario Oficial for 2025, cut from the clause that names the signatory's office, 65.4% of its 33,565 signatures opening with a title (`Lic.`, `Dr.`, `Mtro.`, `C.`, `Licenciada`, `C.P.`), whose first word `parse()` read as a title in 82.1% and `parser_for(locales.ES)` in 86.8% on 2026-10-31. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific titles such as `Guru` in front, `Shekh`, a title there until #752's phase 2j made it one only inside a title phrase, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals --base <base checkout>` the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality; and `local_agreement.py orcid-researchers --base <base checkout>`, ORCID holders' own two boxes (given names, family name) from every country, scored by the country they work or live in -- a population each person split for themselves, academic but worldwide, the one to read for a change a language pack makes; and `local_agreement.py patentsview-inventors --base <base checkout>`, US patents' inventors in the two boxes their filings give, split by a professional filer, with how many people filed under each name in each country, the one to read for how common a name's word is among a country's US-patent inventors. A change to titles or post-nominals, or to the `cs` pack, reads `local_agreement.py cz-candidates --base <base checkout>`, and with `--locale cs` too: Czech election candidates' four boxes (the title before the name, given names, surname, the title after it), scored by election, the only population whose titles are stated apart from the name, 36% of its 170,181 records titled (2026-10-30), nearly all academic degrees; its names are composed from the boxes (`Ing. Jan Novák, Ph.D.`), so it says nothing about how a writer spaces or punctuates them. These are the only real fielded populations outside the committed fields files, and no CI run sees them either.
 
 ## Gotchas
 

tests/v2/test_local_corpora.py

@@ -49,10 +49,10 @@
 # the fetches read, so a patch reaches the code only when it is set on
 # the module holding the name (local_core.RETRIES, local_fec.FEC_MAX_BAD).
 # A second load by path (load_tool) would be a copy nothing reads.
-core, ctgov, fec, tse, tw, ukec, psc, jrc, nepal, orcid, patentsview, volby = (
+core, ctgov, fec, tse, tw, ukec, psc, jrc, nepal, orcid, patentsview, volby, dof = (
     importlib.import_module(f"local_{key}")
     for key in ("core", "ctgov", "fec", "tse", "tw", "ukec", "psc", "jrc", "nepal", "orcid",
-                "patentsview", "volby"))
+                "patentsview", "volby", "dof"))
 
 _SOURCE = "ctgov-officials"
 _DATE = "2026-10-21T09:00:05"
@@ -3347,12 +3347,12 @@ def test_no_module_imports_a_sibling_constant() -> None:
     # module. A `from local_x import CONSTANT` would bind a second name a
     # patch on local_x cannot reach, and a test could pass with its guard
     # off. The modules are SOURCES' own, so a new source is covered. PSC,
-    # JRC, ORCID, PV and VOLBY are sources' names, keys of SOURCES, and no
-    # fetch's setting.
+    # JRC, ORCID, PV, VOLBY and DOF are sources' names, keys of SOURCES, and
+    # no fetch's setting.
     import ast
     sources = {source.fetch.__module__ for source in local.SOURCES.values()}
     assert sources == {m.__name__ for m in (ctgov, fec, tse, tw, ukec, psc, jrc, nepal, orcid,
-                                            patentsview, volby)} | {"local_govuk"}
+                                            patentsview, volby, dof)} | {"local_govuk"}
     siblings = {"local_corpora", "local_core", *sources}
     imported = {(path.stem, alias.name)
                 for path in (Path(str(local.__file__)).parent / f"{s}.py"
@@ -3362,7 +3362,7 @@ def test_no_module_imports_a_sibling_constant() -> None:
                 for alias in node.names if alias.name.lstrip("_").isupper()}
     assert imported == {("local_corpora", "PSC"), ("local_corpora", "JRC"),
                         ("local_corpora", "ORCID"), ("local_corpora", "PV"),
-                        ("local_corpora", "VOLBY")}
+                        ("local_corpora", "VOLBY"), ("local_corpora", "DOF")}
 
 
 def test_the_fold_lists_everything_its_functions_call_and_read() -> None: