From 854bf6c973e23161d30b6b415e7ef2ce3fbab613 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Fri, 30 Oct 2026 18:59:43 -0700 Subject: [PATCH 1/3] Add the dof-signatures local corpus: DOF signature blocks cut to names MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A new unscored local corpus, tools/differential/local_dof.py: the signatures in the 2025 notes of Mexico's Diario Oficial de la Federación, read through SIDOF's web services (a listing per day, each note's HTML; notes robots.txt disallows are skipped). Each name is cut from the clause that signs it -- the last run of bold words with the titles before it, or with no bold the run a title opens -- and the record keeps the clause, the rule, the layout, the title and how the name meets the text before it. Cached and never committed: the signatories are real officials. The fetch caches one gzip of excerpts per day, so it resumes and the cut is re-read offline: DOF_CUT_VERSION re-reads, DOF_EXCERPT_VERSION refetches. It refuses past 5% of stated-size notes coming back blank (image-only notes, which state size 0, are counted apart) and past 10% of "Rúbrica"s left uncut, each uncut one counted by reason. Measured 2026-10-31: 33,623 signatures, 6,663 distinct names, 65.3% opening with a title; 196 of 200 hand-checked cuts exactly a name. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- tests/v2/test_local_corpora.py | 12 +- tests/v2/test_local_dof.py | 323 +++++++++++++++++ tools/differential/README.md | 3 +- tools/differential/local_corpora.py | 15 +- tools/differential/local_dof.py | 532 ++++++++++++++++++++++++++++ 6 files changed, 876 insertions(+), 11 deletions(-) create mode 100644 tests/v2/test_local_dof.py create mode 100644 tools/differential/local_dof.py diff --git a/AGENTS.md b/AGENTS.md index bc923cbc..0f6c07e3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -256,7 +256,7 @@ The 2.0 rewrite landed as underscore-private modules beside the v1 code, which 2 **Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `AFTER_GIVEN_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `nameparser/_reading_index.py`'s `BEHAVIORAL` table, a branch in its `compile_readings` adding the words whose reading carries it and a key in its `Compiled` TypedDict, a constant of the same name in `tools/vocab/build.py`'s `CONSTANTS` roster (an import-time assert there holds the two to one list), and the `NAME = _compiled.NAME` assignment in the config module. A property the compiler reads is kept in the reading index by `indexed()`, which reads `BEHAVIORAL`, so a new one reaches a choice of languages too. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A marker that records a clash with a NAME is the other shape: derived from the word's name reading beside the clashing one (`SUFFIX_ACRONYMS_AMBIGUOUS`, `SUFFIX_WORDS_AMBIGUOUS`, and the complement `NON_GIVEN_NAME_PARTICLES` works from), not written as a property, so it too holds by construction (decisions/vocabulary-model, 2026-10-25). A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). A caller's `Lexicon.add()` to a sub-set does **not** add the word to its base, so a caller adding a brand-new word adds it to both. Two traps a gating set met (#752 phase 2i). Where a caller listing a word in the BASE set has always meant "read it as this everywhere" (the 1.4 recipe `c.titles.add("dean")`), the gated words compile OUT of the base rather than into both, or the gate overrides the caller's listing: `TITLES_AMBIGUOUS` is not a subset of `TITLES` for that reason, and a word in both reads as a title. And `_VOCAB_FIELDS` is read whole by two rosters in `_pipeline/_vocab.py`, the capitals shape's "unlisted" (`in_any_wordlist`) and the comma contrast's "claimed as non-name" (`_NON_NAME_FIELDS`), so a new field joins both by default; a field whose words may be names leaves both, which the grid found as `John Smith, DEAN` reversed where C6 had read a credential; #789's `given_name_abbreviations` met it again (`John Smith, JAS` and `Jas SMITH, XYZ` reversed), found by review because its grid held only the dotted spellings the set was for -- a new field's grid holds the spellings it must NOT move too. -**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials --base `, which reads the base from a baseline kept for its package's files and never the shared snapshot beside the corpus (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating; the shared snapshot beside the corpus is one file per source for every worktree, and #752's phase 2f found ctgov reporting a sibling branch's 1,334 joined-credential readings as its own moves against it); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to English titles, honorifics or post-nominals also reads `snapshot.py --local govuk-people` the same way: GOV.UK's display names of ministers, officials and officers, the title run and the post-nominals as an office writes them (`The Rt Hon`, `Sir`, `Baroness`, ranks; `KC`, `CBE`, `KCB`), in 2,093 of whose 6,021 `parse()` read one or both on 2026-10-30. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific titles such as `Guru` in front, `Shekh`, a title there until #752's phase 2j made it one only inside a title phrase, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals --base ` the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality; and `local_agreement.py orcid-researchers --base `, ORCID holders' own two boxes (given names, family name) from every country, scored by the country they work or live in -- a population each person split for themselves, academic but worldwide, the one to read for a change a language pack makes; and `local_agreement.py patentsview-inventors --base `, US patents' inventors in the two boxes their filings give, split by a professional filer, with how many people filed under each name in each country, the one to read for how common a name's word is among a country's US-patent inventors. A change to titles or post-nominals, or to the `cs` pack, reads `local_agreement.py cz-candidates --base `, and with `--locale cs` too: Czech election candidates' four boxes (the title before the name, given names, surname, the title after it), scored by election, the only population whose titles are stated apart from the name, 36% of its 170,181 records titled (2026-10-30), nearly all academic degrees; its names are composed from the boxes (`Ing. Jan Novák, Ph.D.`), so it says nothing about how a writer spaces or punctuates them. These are the only real fielded populations outside the committed fields files, and no CI run sees them either. +**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials --base `, which reads the base from a baseline kept for its package's files and never the shared snapshot beside the corpus (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating; the shared snapshot beside the corpus is one file per source for every worktree, and #752's phase 2f found ctgov reporting a sibling branch's 1,334 joined-credential readings as its own moves against it); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to English titles, honorifics or post-nominals also reads `snapshot.py --local govuk-people` the same way: GOV.UK's display names of ministers, officials and officers, the title run and the post-nominals as an office writes them (`The Rt Hon`, `Sir`, `Baroness`, ranks; `KC`, `CBE`, `KCB`), in 2,093 of whose 6,021 `parse()` read one or both on 2026-10-30. A change to Spanish titles, to the `es` pack, or to how a title marks where a name starts after an office reads `snapshot.py --local dof-signatures` the same way: the names in the signature blocks of Mexico's Diario Oficial for 2025, cut from the clause that names the signatory's office, 65.3% of its 33,623 signatures opening with a title (`Lic.`, `Dr.`, `Mtro.`, `C.`, `Licenciada`, `C.P.`), whose first word `parse()` read as a title in 82.1% and `parser_for(locales.ES)` in 86.8% on 2026-10-31. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific titles such as `Guru` in front, `Shekh`, a title there until #752's phase 2j made it one only inside a title phrase, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals --base ` the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality; and `local_agreement.py orcid-researchers --base `, ORCID holders' own two boxes (given names, family name) from every country, scored by the country they work or live in -- a population each person split for themselves, academic but worldwide, the one to read for a change a language pack makes; and `local_agreement.py patentsview-inventors --base `, US patents' inventors in the two boxes their filings give, split by a professional filer, with how many people filed under each name in each country, the one to read for how common a name's word is among a country's US-patent inventors. A change to titles or post-nominals, or to the `cs` pack, reads `local_agreement.py cz-candidates --base `, and with `--locale cs` too: Czech election candidates' four boxes (the title before the name, given names, surname, the title after it), scored by election, the only population whose titles are stated apart from the name, 36% of its 170,181 records titled (2026-10-30), nearly all academic degrees; its names are composed from the boxes (`Ing. Jan Novák, Ph.D.`), so it says nothing about how a writer spaces or punctuates them. These are the only real fielded populations outside the committed fields files, and no CI run sees them either. ## Gotchas diff --git a/tests/v2/test_local_corpora.py b/tests/v2/test_local_corpora.py index 4224e714..792d3e87 100644 --- a/tests/v2/test_local_corpora.py +++ b/tests/v2/test_local_corpora.py @@ -49,10 +49,10 @@ # the fetches read, so a patch reaches the code only when it is set on # the module holding the name (local_core.RETRIES, local_fec.FEC_MAX_BAD). # A second load by path (load_tool) would be a copy nothing reads. -core, ctgov, fec, tse, tw, ukec, psc, jrc, nepal, orcid, patentsview, volby = ( +core, ctgov, fec, tse, tw, ukec, psc, jrc, nepal, orcid, patentsview, volby, dof = ( importlib.import_module(f"local_{key}") for key in ("core", "ctgov", "fec", "tse", "tw", "ukec", "psc", "jrc", "nepal", "orcid", - "patentsview", "volby")) + "patentsview", "volby", "dof")) _SOURCE = "ctgov-officials" _DATE = "2026-10-21T09:00:05" @@ -3347,12 +3347,12 @@ def test_no_module_imports_a_sibling_constant() -> None: # module. A `from local_x import CONSTANT` would bind a second name a # patch on local_x cannot reach, and a test could pass with its guard # off. The modules are SOURCES' own, so a new source is covered. PSC, - # JRC, ORCID, PV and VOLBY are sources' names, keys of SOURCES, and no - # fetch's setting. + # JRC, ORCID, PV, VOLBY and DOF are sources' names, keys of SOURCES, and + # no fetch's setting. import ast sources = {source.fetch.__module__ for source in local.SOURCES.values()} assert sources == {m.__name__ for m in (ctgov, fec, tse, tw, ukec, psc, jrc, nepal, orcid, - patentsview, volby)} | {"local_govuk"} + patentsview, volby, dof)} | {"local_govuk"} siblings = {"local_corpora", "local_core", *sources} imported = {(path.stem, alias.name) for path in (Path(str(local.__file__)).parent / f"{s}.py" @@ -3362,7 +3362,7 @@ def test_no_module_imports_a_sibling_constant() -> None: for alias in node.names if alias.name.lstrip("_").isupper()} assert imported == {("local_corpora", "PSC"), ("local_corpora", "JRC"), ("local_corpora", "ORCID"), ("local_corpora", "PV"), - ("local_corpora", "VOLBY")} + ("local_corpora", "VOLBY"), ("local_corpora", "DOF")} def test_the_fold_lists_everything_its_functions_call_and_read() -> None: diff --git a/tests/v2/test_local_dof.py b/tests/v2/test_local_dof.py new file mode 100644 index 00000000..98c7eed5 --- /dev/null +++ b/tests/v2/test_local_dof.py @@ -0,0 +1,323 @@ +"""Offline checks on tools/differential/local_dof.py, the dof-signatures +local corpus. + +Every name here is invented: the signatories in the DOF are real +officials, and their names stay in the cache. What these pin is the +reading of a note's HTML (blocks and bold), the cut rules and what each +records, the fetch's walk over days (resumed from the cache, the notes +robots.txt disallows skipped, a failed request retried) and its +refusals. +""" +import datetime +import gzip +import importlib +import json +import time +import urllib.error +import urllib.request +from collections.abc import Callable +from pathlib import Path +from typing import Any + +import pytest + +from ._differential_fixtures import _Response, load_tool, own_cache + +local = load_tool("local_corpora") +dof = importlib.import_module("local_dof") + +_DOF = "dof-signatures" +_BOLD = "{}" + + +@pytest.fixture(autouse=True) +def _own_cache(monkeypatch: pytest.MonkeyPatch, tmp_path_factory: pytest.TempPathFactory) -> Path: + return own_cache(monkeypatch, tmp_path_factory) + + +def _div(text: str) -> str: + return f"
{text}
" + + +#: A court notice, stacked: date, office, the name in bold, "Rúbrica." +#: alone; then an agreement, inline, two signatures in one paragraph, +#: the second name before its office; and a signature in capitals. +_STACKED = (_div("Texto del edicto.") + _div("Ciudad de México, a 2 de enero de 2025.") + + _div("La Secretaria del Juzgado Primero de Distrito") + + f"
{_BOLD.format('Lic. Ana Ruiz Toledo')}
" + _div("Rúbrica.")) +_INLINE = ("

Se firma en la Ciudad de México, a 2 de enero de 2025.- Por la Entidad: " + "el Director General, Mtra. " + _BOLD.format("Eva") + " " + _BOLD.format("Mar Soto") + + ".- Rúbrica.- " + _BOLD.format("Bo Gil Paz") + ", Presidente del Comité.- " + "Rúbrica.

") +_CAPITALS = (_div("JEFE DE DEPARTAMENTO") + f"
{_BOLD.format('ING. LUIS PRADO REY')}
" + + _div("Rúbrica.")) + + +def _records(content: str) -> list[dict[str, str] | str]: + return [found for excerpt in dof.excerpts(content) for found in dof.signatures(excerpt)] + + +def test_a_stacked_signature_cuts_the_bold_block_and_keeps_the_office_above() -> None: + assert _records(_STACKED) == [{ + "name": "Lic. Ana Ruiz Toledo", "cut": "bold", "layout": "stacked", + "title": "Lic.", "joined": "block", + "clause": "Ciudad de México, a 2 de enero de 2025. | La Secretaria del Juzgado " + "Primero de Distrito | Lic. Ana Ruiz Toledo"}] + + +def test_an_inline_signature_cuts_the_bold_run_with_the_title_before_it() -> None: + first, second = _records(_INLINE) + assert first == { + "name": "Mtra. Eva Mar Soto", "cut": "bold", "layout": "inline", "title": "Mtra.", + "joined": "comma", + "clause": "Por la Entidad: el Director General, Mtra. Eva Mar Soto"} + # The name before its office: the bold run ends at the comma. + assert second == { + "name": "Bo Gil Paz", "cut": "bold", "layout": "inline", "title": "", + "joined": "start", "clause": "Bo Gil Paz, Presidente del Comité"} + + +def test_a_title_written_in_capitals_is_read_as_one() -> None: + (found,) = _records(_CAPITALS) + assert isinstance(found, dict) + assert (found["name"], found["title"]) == ("ING. LUIS PRADO REY", "ING.") + + +@pytest.mark.parametrize(("clause", "expected"), [ + # No bold: the run from the last title to the end, the office before + # it glued with no comma. + ("El Secretario del Juzgado Licenciada Ema Rey Gil", ("Licenciada Ema Rey Gil", "title")), + ("Directora de Área, la C.P. Ana Lis Mora", ("C.P. Ana Lis Mora", "title")), + # The office after the name, set off by a comma, stays out. + ("Sen. Ana Lis Mora, Secretaria", ("Sen. Ana Lis Mora", "title")), +]) +def test_without_bold_a_title_opens_the_cut(clause: str, expected: tuple[str, str]) -> None: + assert dof.cut(clause, "0" * len(clause)) == expected + + +@pytest.mark.parametrize("clause", [ + "El Titular de la Unidad, la Ana Lis Mora", # neither bold nor a title + "Ana Lis Mora", + "II. Remitir mediante oficio dirigido a la Dirección General el convenio", # a sentence +]) +def test_without_bold_or_a_title_short_enough_nothing_is_cut(clause: str) -> None: + assert dof.cut(clause, "0" * len(clause)) is None + + +def test_a_title_in_roman_after_a_bold_office_stays_out_of_an_earlier_run() -> None: + # The bold office ends in a comma, so it is a run of its own, and + # the last run is the name. + text = "Gerente de Área, Lic. Ana Lis Mora" + mask = "".join("1" if 13 <= i <= 15 or i >= 21 else "0" for i in range(len(text))) + assert dof.cut(text, mask) == ("Lic. Ana Lis Mora", "bold") + + +@pytest.mark.parametrize(("before", "joined"), [ + ("", "start"), ("Oficina | ", "block"), ("Director General, ", "comma"), + ("Por la Entidad: ", "colon"), ("Directora, la ", "article"), + ("La Secretaria del Juzgado ", "word")]) +def test_how_a_cut_meets_the_clause_before_it(before: str, joined: str) -> None: + assert dof.joined(before) == joined + + +@pytest.mark.parametrize("word", ["Lic.", "C.", "C.P.", "L.C.P.", "Mtro.", "Lcda.", "LIC.", + "Licenciado", "MAESTRA"]) +def test_titles_by_shape_and_by_spelling(word: str) -> None: + assert dof.is_title(word) + + +@pytest.mark.parametrize("word", ["Ana", "Rúbrica.", "de", "Lic", "CONSEJO."]) +def test_words_that_are_not_titles(word: str) -> None: + assert not dof.is_title(word) + + +def test_a_rubrica_with_nothing_before_it_is_counted_not_cut() -> None: + assert _records(_div("Rúbrica.")) == ["stacked, no block above"] + assert _records("

firman de conformidad, .- Rúbrica.

") == ["inline, nothing before"] + + +@pytest.mark.parametrize(("name", "reason"), [ + ("a siete de noviembre de dos mil veinticinco", "cut a date"), + ("Ciudad de México a 2 de enero de 2025", "cut a date"), + ("EL SECRETARIO DE SALUD", "cut an office"), + ("La Gobernadora Constitucional", "cut an office"), + ("Atentamente", "cut one word"), +]) +def test_a_cut_of_one_word_a_date_or_an_office_is_counted_not_kept(name: str, reason: str) -> None: + signed = _div("Atentamente") + f"
{_BOLD.format(name)}
" + _div("Rúbrica.") + assert _records(signed) == [f"stacked, {reason}"] + + +@pytest.mark.parametrize("name", [ + "Lic. Julio Linares Rangel", "Del Olmo Blanco", "CAP. 1/O. AYTE. LUIS REY PAZ"]) +def test_a_month_a_digit_or_del_inside_a_name_is_kept(name: str) -> None: + signed = _div("Atentamente") + f"
{_BOLD.format(name)}
" + _div("Rúbrica.") + assert [r["name"] for r in _records(signed) if isinstance(r, dict)] == [name] + + +def test_adjacent_stacked_signatures_share_an_excerpt_and_cut_apart() -> None: + two = _STACKED + _div("El Actuario") + f"
{_BOLD.format('Juan Rey Paz')}
" \ + + _div("Rúbrica.") + assert len(dof.excerpts(two)) == 1 + names = [r["name"] for r in _records(two) if isinstance(r, dict)] + assert names == ["Lic. Ana Ruiz Toledo", "Juan Rey Paz"] + assert _records(two)[1]["clause"] == "El Actuario | Juan Rey Paz" # type: ignore[index] + + +def test_robots_disallows_only_the_notes_listed_for_every_agent() -> None: + robots = ("User-agent: *\nDisallow: /CopiasCert?\nDisallow: /notas/5\n" + "Disallow: /notas/docFuente/6\n\nUser-agent: AdsBot-Google\nDisallow: /notas/7\n") + assert dof.disallowed(robots) == frozenset({5, 6}) + + +# -- the fetch ----------------------------------------------------------------- + +_DAY1, _DAY2 = datetime.date(2025, 1, 2), datetime.date(2025, 1, 3) + + +def _listing(*entries: dict[str, Any]) -> dict[str, Any]: + return {"messageCode": 200, "response": "OK", "NotasMatutinas": list(entries), + "NotasVespertinas": [], "NotasExtraordinarias": []} + + +def _entry(cod: int, html: str = "S") -> dict[str, Any]: + return {"codNota": cod, "existeHtml": html} + + +#: Day 1: the stacked note, the inline note, a section heading without +#: HTML and a note robots.txt disallows. Day 2: no edition. +_NOTES: dict[int, str | None] = {1: _STACKED, 2: _INLINE, 9: _CAPITALS} +_DAYS = {"02-01-2025": _listing(_entry(1), _entry(2), _entry(3, "N"), _entry(9)), + "03-01-2025": _listing()} + + +def _serve(monkeypatch: pytest.MonkeyPatch, *, + days: dict[str, dict[str, Any]] = _DAYS, notes: dict[int, str | None] = _NOTES, + fail: Callable[[str], Exception | bytes | None] = lambda url: None) -> list[str]: + """Answer SIDOF from `days` and `notes`, robots.txt disallowing note + 9, and a note of None as one published as images, stating size 0; + `fail` may answer a URL instead. Returns the URLs fetched.""" + monkeypatch.setattr(dof, "DOF_FROM", _DAY1) + monkeypatch.setattr(dof, "DOF_TO", _DAY2) + monkeypatch.setattr(time, "sleep", lambda seconds: None) + asked: list[str] = [] + + def urlopen(request: urllib.request.Request, timeout: int) -> _Response: + url = request.full_url + asked.append(url) + answer = fail(url) + if isinstance(answer, Exception): + raise answer + if isinstance(answer, bytes): + return _Response(answer) + if url == dof.DOF_ROBOTS_URL: + return _Response(b"User-agent: *\nDisallow: /notas/9\n") + tail = url.rsplit("/", 1)[1] + if "/nota/" in url: + content = notes[int(tail)] + note = ({"cadenaContenido": content} if content is not None + else {"cadenaContenido": None, "tamanioContenido": "0"}) + body: object = {"messageCode": 200, "Nota": note} + else: + body = days[tail] + return _Response(json.dumps(body).encode()) + monkeypatch.setattr(urllib.request, "urlopen", urlopen) + return asked + + +def test_a_dof_fetch_walks_the_days_and_keeps_each_signature( + monkeypatch: pytest.MonkeyPatch) -> None: + asked = _serve(monkeypatch) + corpus = local.fetch(_DOF) + assert [(r["name"], r["day"], r["note"]) for r in corpus.records] == [ + ("Lic. Ana Ruiz Toledo", "2025-01-02", "1"), ("Mtra. Eva Mar Soto", "2025-01-02", "2"), + ("Bo Gil Paz", "2025-01-02", "2")] + assert dof.DOF_NOTE_URL.format(cod=9) not in asked + assert corpus.meta["data_date"] == "2025-01-03" + detail = corpus.meta["detail"] + assert detail["notes"] == {"read": 2, "without html": 1, "disallowed": 1} + assert (detail["cuts"], detail["uncut"]) == ({"bold": 3}, {}) + assert corpus.header().endswith("3 signatures (local corpus dof-signatures)") + + +def test_a_second_fetch_reads_the_cached_days_and_asks_only_robots( + monkeypatch: pytest.MonkeyPatch) -> None: + _serve(monkeypatch) + first = local.fetch(_DOF).records + asked = _serve(monkeypatch) + assert local.fetch(_DOF).records == first + assert asked == [dof.DOF_ROBOTS_URL] + + +def test_a_day_cached_under_another_excerpt_version_is_fetched_again( + monkeypatch: pytest.MonkeyPatch) -> None: + _serve(monkeypatch) + local.fetch(_DOF) + path = dof._day_path(_DAY1) + held = json.loads(gzip.decompress(path.read_bytes())) + path.write_bytes(gzip.compress(json.dumps({**held, "excerpt_version": 0}).encode())) + asked = _serve(monkeypatch) + local.fetch(_DOF) + assert dof.DOF_DAY_URL.format(day="02-01-2025") in asked + assert dof.DOF_DAY_URL.format(day="03-01-2025") not in asked + + +def test_a_damaged_cached_day_is_refused_naming_the_way_out( + monkeypatch: pytest.MonkeyPatch) -> None: + _serve(monkeypatch) + local.fetch(_DOF) + dof._day_path(_DAY1).write_bytes(b"not a gzip") + with pytest.raises(ValueError, match="delete it and fetch again"): + local.fetch(_DOF) + + +def test_a_dof_request_failing_once_is_retried(monkeypatch: pytest.MonkeyPatch) -> None: + note = dof.DOF_NOTE_URL.format(cod=2) + queue: list[Exception | bytes | None] = [urllib.error.URLError("reset"), b"{\"cut"] + asked = _serve(monkeypatch, fail=lambda url: queue.pop(0) if url == note and queue else None) + assert len(local.fetch(_DOF).records) == 3 + assert asked.count(note) == 3 + + +def test_an_answer_that_is_not_message_code_200_is_refused( + monkeypatch: pytest.MonkeyPatch) -> None: + day = dof.DOF_DAY_URL.format(day="02-01-2025") + _serve(monkeypatch, fail=lambda url: b'{"messageCode": 500}' if url == day else None) + with pytest.raises(ValueError, match="answered 500"): + local.fetch(_DOF) + + +def test_notes_that_came_back_empty_past_the_gate_refuse_the_fetch( + monkeypatch: pytest.MonkeyPatch) -> None: + _serve(monkeypatch, notes={**_NOTES, 2: ""}) + with pytest.raises(ValueError, match="1 of 2 notes with HTML came back empty"): + local.fetch(_DOF) + monkeypatch.setattr(dof, "DOF_MAX_EMPTY", 0.5) + assert local.fetch(_DOF).meta["detail"]["notes"]["empty"] == 1 + + +def test_a_note_published_as_images_is_counted_apart_and_passes_the_gate( + monkeypatch: pytest.MonkeyPatch) -> None: + _serve(monkeypatch, notes={**_NOTES, 2: None}) + detail = local.fetch(_DOF).meta["detail"] + assert detail["notes"] == {"read": 1, "image only": 1, "without html": 1, "disallowed": 1} + + +def test_rubricas_no_rule_cut_past_the_gate_refuse_the_fetch( + monkeypatch: pytest.MonkeyPatch) -> None: + _serve(monkeypatch, notes={**_NOTES, 2: _div("Rúbrica.") * 2}) + with pytest.raises(ValueError, match="2 of 3 Rúbricas cut no name"): + local.fetch(_DOF) + + +def test_a_corpus_cut_by_another_version_is_refused_on_load( + monkeypatch: pytest.MonkeyPatch) -> None: + _serve(monkeypatch) + local.fetch(_DOF) + assert local.load(_DOF) is not None + monkeypatch.setattr(dof, "DOF_CUT_VERSION", dof.DOF_CUT_VERSION + 1) + with pytest.raises(local.CorpusDamaged, match="cut by cut version"): + local.load(_DOF) + monkeypatch.setattr(dof, "DOF_TO", datetime.date(2025, 1, 4)) + with pytest.raises(local.CorpusDamaged, match="fetched to 2025-01-03"): + local.load(_DOF) diff --git a/tools/differential/README.md b/tools/differential/README.md index d7be0780..f8daeffe 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -263,7 +263,7 @@ uv run python tools/differential/build_shapes_corpus.py --coverage A local corpus holds real names that are not committed: fetched into the cache by `local_corpora.py`, read by `snapshot.py --local` and by a row of `tools/name_usage.py`, scored by `local_agreement.py` where its records carry their fields, and seen by no reviewer and no CI run (the SURVEY job in mechanisms.md#NAME-DATA-JOBS, and WATCH and SCORE outside CI). What reaches the repository is what a contributor cites from it, counts and shapes written with made-up names. -`local_corpora.py` holds the `SOURCES` registry, the cache's files, the views and the command line; each source's fetch is a module of its own (`local_ctgov.py`, `local_fec.py`, `local_tse.py`, `local_tw.py`, `local_ukec.py`, `local_govuk.py`, `local_psc.py`, `local_jrc.py`, `local_nepal.py`, `local_orcid.py`, `local_patentsview.py`, `local_volby.py`) whose docstring says what the source is and what its terms allow, built on the retry policy, share gate and downloads in `local_core.py`. A new source is a module and a `SOURCES` row. A source whose records hold a value its own code derives, rather than one the source gave (`jrc-names`' script label), stores a read version in `meta.json`'s detail and refuses on load a corpus read by another (`JRC_READ_VERSION`, bumped by hand with any change to how a record is built; `psc-individuals` does the same for its cached extracts, with `PSC_EXTRACT_VERSION` in each extract's file name): the sha256 checks prove the files are the ones fetched, not that today's code would write them. A constant such as `TSE_MAX_BAD` or `RETRIES` lives in one module, which every reader reads it through, so a test patches it there; no other module `from`-imports it, so a patch set elsewhere raises rather than missing the code (`test_no_module_imports_a_sibling_constant` holds this). +`local_corpora.py` holds the `SOURCES` registry, the cache's files, the views and the command line; each source's fetch is a module of its own (`local_ctgov.py`, `local_fec.py`, `local_tse.py`, `local_tw.py`, `local_ukec.py`, `local_govuk.py`, `local_psc.py`, `local_jrc.py`, `local_nepal.py`, `local_orcid.py`, `local_patentsview.py`, `local_volby.py`, `local_dof.py`) whose docstring says what the source is and what its terms allow, built on the retry policy, share gate and downloads in `local_core.py`. A new source is a module and a `SOURCES` row. A source whose records hold a value its own code derives, rather than one the source gave (`jrc-names`' script label), stores a read version in `meta.json`'s detail and refuses on load a corpus read by another (`JRC_READ_VERSION`, bumped by hand with any change to how a record is built; `psc-individuals` does the same for its cached extracts, with `PSC_EXTRACT_VERSION` in each extract's file name): the sha256 checks prove the files are the ones fetched, not that today's code would write them. A constant such as `TSE_MAX_BAD` or `RETRIES` lives in one module, which every reader reads it through, so a test patches it there; no other module `from`-imports it, so a patch set elsewhere raises rather than missing the code (`test_no_module_imports_a_sibling_constant` holds this). ``` uv run python tools/differential/local_corpora.py fetch ctgov-officials @@ -304,6 +304,7 @@ Any snapshot reading of 5,000 entries or more, committed or local, parses in a w | `jrc-names` | the person entities of the EU Joint Research Centre's JRC-Names, each with the spellings of its name harvested from news text in many languages and scripts: a record is one distinct spelling, with its entity and its script. The file (`entities.gzip`, 14.8 MB) is pinned by sha256 and dated by its Last-Modified, 2024-03-15. Of its 1,317,586 spellings kept, 1,203,634 are Latin, 66,673 Cyrillic, 17,074 Arabic (7,833 persons, 5,347 of them spelled in Latin too), 13,039 Katakana and 8,016 Han, and 218,822 persons have two or more spellings, 25,415 of them in two scripts or more (a spelling whose letters are in two scripts, or none, not counted as one). The population for spelling and transliteration variants of one name, read by the `groups` view, and for Arabic-script names. Licensed under the JRC-Names EULA, personal and non-transferable, with no redistribution or derivative works, so its views' output and its snapshot stay in the cache like every local corpus's, and here the licence, not only the names being real people's, is why | titles and credentials, since the names are harvested from running text (`parse()` read a title in 3,011 of its 1,315,051 names and a suffix in 1,940, 2026-10-24); the ASCII comma, which none holds (1,510 Arabic spellings hold an Arabic comma, U+060C, which the parser reads as one); the Indic scripts (Devanagari 827 spellings, Bengali 419, the others fewer); people the news did not name | | `nepal-candidates` | every candidate in Nepal's local elections of 2079 BS (2022), from the Election Commission's results site (result.election.gov.np), one file per local body: a record per distinct pair of the Commission's romanization (`name`, the one `parse()` reads) and the name in Devanagari as filed (`devanagari`). The corpus and the word index hold both, so a snapshot and a `name_usage.py` lookup read both scripts; `tally`, which counts what follows a comma, reads the romanized name. Of 126,981 pairs fetched 2026-10-24 (152,960 candidates in 753 files, 26 of whose romanizations were a placeholder such as `- - -` and are dropped), the romanized names carry surnames written as initials (`B.K.`, `K.C.`, `G.C.`), honorific words that are titles in front (`Guru`; `Shekh` until #752's phase 2j made it a title only in a title phrase) and `Md.`, Muhammad's abbreviation, which its period makes a title by shape; the Devanagari ones carry vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and the joiners ZWJ and ZWNJ. The population for romanized South Asian names and for Devanagari. The site states no licence, which the credit line says | case and the comma, since the Commission's romanization is one consistent hand; names a person typed themselves; credentials and generational words; the parliament and provincial candidates, whose files carry Devanagari only; names outside Nepal | | `cz-candidates` | every candidate on the registers the Czech Statistical Office (ČSÚ) publishes on volby.gov.cz for the 2022 municipal elections (`kv2022`) and the 2025 Chamber of Deputies election (`ps2025`): a record is a distinct candidate's four name boxes in one election, the title before the name (`TITULPRED`), the given names (`JMENO`), the surname (`PRIJMENI`) and the title after it (`TITULZA`), with the election; 165,736 records of kv2022's 197,506 candidates and 4,445 of ps2025's 4,475 on 2026-10-30. A record's name is COMPOSED, not written by anyone: the boxes joined in the Czech convention, `TITULPRED JMENO PRIJMENI, TITULZA` (`Ing. Jan Novák, Ph.D.`). 61,320 of its 170,181 records carry a title (58,667 before the name, 7,687 after it), nearly all academic degrees (`Ing.`, `Mgr.`, `Bc.`, `MUDr.`; `DiS.`, `Ph.D.`, `MBA`): the population for titles and post-nominals the filer kept apart from the name. Scored by `local_agreement.py` by election, under the default parser and under `--locale cs`. ČSÚ licenses it under CC BY 4.0, asking that derived data be marked as derived, which the composed names are and the credit line says | case, spacing and punctuation of a writer's own, since the string is composed; the family-comma shape; courtesy titles and offices; names outside Czechia; the elections not in `VOLBY_ELECTIONS` (the index lists kv2002 to kv2026 and ps2002 to ps2025) | +| `dof-signatures` | the signatures in the notes of Mexico's Diario Oficial de la Federación (DOF) for 2025, read through SIDOF's web services (a listing per day, each note's HTML; the notes its robots.txt disallows are skipped), each name cut from the clause that signs it (`local_dof.cut`): the last run of bold words, with the titles before it, which the DOF mostly sets in roman, or with no bold the run a title opens, up to a comma. A record is one signature (a signatory signs many notes): the name, the rule that cut it, the layout (`stacked`, the office on lines above the name and `Rúbrica.` on a line below; `inline`, `Office, Title Name.- Rúbrica.-` in a paragraph), the title the name opens with, how it meets the text before it, and the clause. Of the 35,305 `Rúbrica`s in 2025's notes (fetched 2026-10-31: of the 26,752 notes listed with HTML, 2,936 were published as page images, stating a size of 0, and 22 came back blank though they stated one), 33,623 were cut, to 6,663 distinct names, and the 1,682 left are counted by reason in `meta.json` (neither bold nor a title, an office signing without a name, a date, one word). Of the signatures, 65.3% open with a title (`Lic.` 27.7%, `Dr.`/`Dra.` 9.3%, `Mtro.`/`Mtra.` 8.1%, `C.` 5.1%, `Ing.` 3.6%, a spelled-out `Licenciado`/`Licenciada` 2.9%, a dotted acronym degree such as `C.P.` or `L.C.P.` in most of the rest), and 45.4% stand under their office in a stacked block, glued to it with no comma once the lines are read as one. Of 200 cuts drawn at random and checked by hand on 2026-10-31 (cut version 3), 196 were exactly a name, 3 a name with an office or degree (`Juez`, `Magistrado`, `Doctora en Derecho.`) and 1 not a name (`f`, a one-word cut refused since). The population for Spanish titles in front of a name. The DOF's texts are not protected by copyright (LFDA art. 14 VIII), and SIDOF permits reproduction without profit with the source cited, which the credit line says | the office before the name, which the cut drops, so a snapshot reads names with nothing before them, and the comma; names a person typed themselves, since the DOF's typesetters write every one; private persons, a signatory being an official, a judge or a party's representative; names outside Mexico and in any script but Latin. The `title` field reads a title by shape, so a lone initial counts as one (`J. Guadalupe`, José's) | A live source has no fixed snapshot to pin, so the cached copy is the pin: `meta.json` holds the source's data date and the sha256 of `raw.jsonl` and `corpus.jsonl`, checked on every load, and every report, the `name_usage.py` row included, carries the source and that date. Quote a count with the date, and expect a later fetch to differ. A local snapshot records which fetch it was read from, and `snapshot.py --local` refuses to compare it with a corpus fetched since, which would report the names the source gained and lost as moves: after a `fetch --refresh`, record again from the base checkout. ClinicalTrials.gov's terms ask anyone redistributing its data to credit it, show NLM's processing date and say what was changed; keeping the names in the cache and citing counts with that header meets them. The FEC's bar on soliciting and commercial use is in the credit line of every `fec-individuals` report, so a count quoted from it carries the bar with it. A record there is a distinct name, not a contribution or a contributor: one donor gives many times (34,264,742 rows held 1,893,964 names in the file dated 2026-10-18), and counting contributions would weight a shape by how often its bearers give. `tools/name_usage.py` looks a word up in each corpus's word index, cached as `wordindex.json` beside it on first use and keyed to the records' sha256, to the code that cuts and folds a word and to the record keys it reads (`Source.corpus_keys`; a record whose two names write one word counts once), so a refetch, a change to the fold or a change to which of a record's names are read rebuilds it; a single-word lookup it answers parses neither the records nor the names. Measured 2026-10-22 with both corpora fetched, a single-word `name_usage.py --no-wikidata` run took 2.5s with their indexes cached (12.5s before the cache, 14.8s on the run that builds it). A spaced word (`Ph. D.`) still scans every name, which took 8.6s, nearly all of it the FEC corpus's. For `orcid-researchers` and `patentsview-inventors`, whose extracts keep every record's boxes and country uncapped, `name_usage.py --boxes` adds a row by box and country, read from a second index, `boxindex.json.gz` beside the corpus (`cached_box_index`): per word and country the people whose given box or family box writes it, each country shown as a share of its people, so a word can be weighed in the country a language pack is for (`--country CC`). It is keyed to the extract's sha256, taken from the bytes as they are read, and to a fingerprint of the reader and the fold, so a refetch rebuilds it. Building it took 2 min 19 s for ORCID's 24,990,909 records (3,780,309 words in 6,660,756 word-and-country cells, 41 MB cached) on 2026-10-28. With both indexes cached a lookup took 10.4 s with `--boxes` against 5.3 s without, the difference nearly all reading the two indexes (the 369 MB extract hashes in a fraction of a second once the system has it cached), which is why the rows are asked for rather than always built; an extract that cannot be read fails its own row, naming the refresh. `fetch --refresh` re-downloads the FEC's zip rather than re-reading the cached one, replacing it only once the new one is whole, and a corpus fetched for another cycle than `FEC_CYCLE` is refused on load. The `fec-individuals` fetch refuses a file whose lines stop matching the 21 columns it reads, or whose names stop holding a comma, rather than read it as a smaller or a different corpus. A `ukec-individuals` record is a distinct name too, not a donation or a `DonorId`: one name often holds several IDs, which seem to come with reports rather than with people (15,214 names held 24,473 on 2026-10-22, the IDs counted in `meta.json`'s detail; 2,663 names held several at one `RegulatedEntityId` alone, counted from the export and not kept, though some of those may be two people of one name), so a count of IDs would weight a name by how often it was reported. Its fetch reads the register's total from the JSON search under the same filter before and after the export, and refuses an export holding another number of rows, or a register that changed in between: the export ignores `rows=`, so the total is what tells a cut or capped export from the whole register. It refuses an export whose last row is cut short (Python's `csv` returns a body cut inside a row as a short last row, which the row count would otherwise take for a whole one), and a header other than the 29 columns `UKEC_COLUMNS` names, and counts into `meta.json`'s detail the rows it skips: rows not 29 fields, rows from a donor that is not an individual (the filter asks the server to leave those out), and individuals with no name. Past `UKEC_MAX_BAD` of either of the first two, or `UKEC_MAX_NAMELESS` of the third (4 of 42,355 rows were nameless on 2026-10-22), it refuses the export. The data date is when the first search answered, in UTC: the register states no date of its own, so the counts above, dated by the day they were fetched (2026-10-22), carry the data date 2026-10-23T03:13:19Z in their header. `tw-donors` keeps one zip per report under `reports/`, whole, its masked ID, address and phone columns included, as the FEC's zip is kept: the platform's download answers a report that does not exist and a fault alike, with a 500 and an incident number, so the fetch reads which reports exist from the platform's listing and refuses a listed report it cannot download. It reads each file's columns by name, since a 2020 report lacks a column the newest ones write, and refuses a column it does not know, a category it does not know, a row whose serial column names another report (the zip is not the one asked for), a listing that changes during the fetch, and a report past `TW_MAX_REPORT_BAD` of whose rows it reads past. It reads back the rows a stray quote took into one field (169 rows in 12 records of three reports on 2026-10-22, 8 of them two rows joined by two stray quotes), counts donor cells that hold no name (匿名, a dash) or a masked one, and caches a downloaded zip only once its rows are read, so a refused zip is asked for again. A zip the platform rebuilt under the name it had is not seen: delete `reports/` to download every report again. A layout check written from one surveyed file meets every other file only at fetch time: `tw-donors`' first three fetches were each refused partway, by a report with no income writing one row of dashes, by a 2020 report with 24 columns where the surveyed one had 25, and by a row with no category. Each refusal cost every download again until the source cached its zips, so a new source that downloads many files caches each one before its checks run. `tse-donors` caches nothing but the corpus: each election's national member (`receitas_candidatos__BRASIL.csv`) is read from the TSE's CDN in HTTP Range requests, 209 MB for the two elections where their zips are 1.82 GB (2026-10-22), because the TSE regenerates the zips every few days and a cached one would be stale within the week. Every range request carries `If-Range` with the ETag the HEAD gave if it is strong, else its `Last-Modified`, and every part must state the `Content-Range` asked for, so a zip replaced during the read is refused rather than read across two versions; a HEAD that states no length, neither a strong ETag nor a `Last-Modified`, or `Accept-Ranges` other than `bytes` is refused before the read. A fetch took 69s on 2026-10-22. It refuses a zip without the national member, a header that is not 60 columns or lacks its three columns, a `DS_ORIGEM_RECEITA` value it does not know (a renamed category, or the file no longer in Latin-1), malformed rows or lines folded into a field past `TSE_MAX_BAD`, an election with too few rows from individuals (`TSE_MIN_INDIVIDUALS`: a member cut to its header, or the donations moved under another category) too many without a name or registry name (`TSE_MAX_MISSING`, counting a placeholder such as `#NULO` or `***` as none: a masked column) or with a number in one (`TSE_MAX_NUMBERED`: a numeric column moved under a name), too few distinct filed or registry names (`TSE_MIN_DISTINCT`: a column masked with one word), and a corpus fetched for other elections than `TSE_ELECTIONS`. The CPF in each row is unmasked in the file, and like every other column it is read past, never stored or printed; a pair either of whose names holds more than `TSE_MAX_DIGITS` digits is dropped and counted, since campaigns sometimes typed a number, a CPF among them, into the name field. A record is a pair, so the counts in its reports (`tally`, the `name_usage.py` row) are of pairs, and a filed name standing beside two registry names counts twice. diff --git a/tools/differential/local_corpora.py b/tools/differential/local_corpora.py index 5707a1d8..2ff42a00 100644 --- a/tools/differential/local_corpora.py +++ b/tools/differential/local_corpora.py @@ -47,7 +47,8 @@ name in a country, its two boxes joined, with the boxes, the country and how many people filed under it; one distinct Czech election candidate's four boxes composed into a name, with the boxes and the - election. The counts are counted from here. + election; one DOF signature, the name cut from its clause, with the + clause, the cut rule and the note. The counts are counted from here. - corpus.jsonl -- the distinct names, in the corpus line format snapshot.py and compare.py --corpus read: each record's "name", and for nepal-candidates its Devanagari name too, and for @@ -86,8 +87,8 @@ local_govuk.py (govuk-people), local_psc.py (psc-individuals), local_jrc.py (jrc-names), local_nepal.py (nepal-candidates), local_orcid.py -(orcid-researchers), local_patentsview.py (patentsview-inventors) and -local_volby.py (cz-candidates). +(orcid-researchers), local_patentsview.py (patentsview-inventors), +local_volby.py (cz-candidates) and local_dof.py (dof-signatures). """ from __future__ import annotations @@ -124,6 +125,8 @@ import local_core # noqa: E402 from local_core import Fetched as Fetched, Source, _again, fetch_command # noqa: E402 from local_ctgov import fetch_ctgov # noqa: E402 +import local_dof # noqa: E402 +from local_dof import DOF, check_dof, fetch_dof, renew_dof # noqa: E402 import local_fec # noqa: E402 import local_tse # noqa: E402 from local_fec import check_fec, fetch_fec, renew_fec # noqa: E402 @@ -267,6 +270,12 @@ def index(self) -> WordIndex: "zverejnovani_statistickych_udaju_csu; each name composed from " "the filed boxes, so derived data)", unit="records", fetch=fetch_volby, check=check_volby), + DOF: Source( + credit=f"Diario Oficial de la Federación (Mexico), signatures in the notes " + f"of {local_dof.DOF_FROM} to {local_dof.DOF_TO}, read through SIDOF's " + f"web services (reproduction without " + "profit permitted with the source cited: sidof.segob.gob.mx/avisoLegal)", + unit="signatures", fetch=fetch_dof, renew=renew_dof, check=check_dof), } diff --git a/tools/differential/local_dof.py b/tools/differential/local_dof.py new file mode 100644 index 00000000..e38b1ea6 --- /dev/null +++ b/tools/differential/local_dof.py @@ -0,0 +1,532 @@ +"""The dof-signatures local corpus: the signatures in Mexico's Diario +Oficial de la Federación. + +dof-signatures -- one record per signature in the notes the DOF published +from DOF_FROM to DOF_TO, each the name cut from the clause that precedes +its "Rúbrica" (the printed mark of a signed original), with the clause +kept beside it. The signatories are officials, judges' clerks and the +parties to agreements, and they sign as the DOF prints them: in spoken +order, mixed case, mostly with the courtesy or academic title they use +(`Lic.`, `Mtra.`, `C.`, `Dr.`, `Ing.`, `Licenciada`), after an office +that a comma usually, but not always, sets off. So it is the population +for Spanish titles as written, and for a title that marks where a name +starts after an office. Unscored: no box says which words are the name, +so the name is a cut, and the cut is a rule this module states (cut()) +and records on every row, which a reader can filter by. + +Fetched through SIDOF, the DOF's own site, from the JSON web services +its open-data page offers (sidof.segob.gob.mx/datos_abiertos): a day's +listing (`/dof/sidof/notas/`), then each note's HTML +(`/dof/sidof/notas/nota/`), sequentially, DOF_DELAY seconds +apart, skipping the notes robots.txt disallows. Each day's excerpts -- +the blocks around each "Rúbrica", with which characters the DOF set in +bold -- are cached beside the corpus under days/, written once the +day's every note has been read, so a fetch cut short resumes at the +first day it has not cached and the cut can be read again without +fetching anything (three days sampled on 2026-10-30 ran to 3, 12 and +18 MB of HTML, where a day's excerpts are under 10 KB gzipped). + +Terms: the Ley Federal del Derecho de Autor, art. 14 fr. VIII, puts +"Los textos legislativos, reglamentarios, administrativos o judiciales" +outside copyright, and SIDOF's Aviso Legal (sidof.segob.gob.mx/avisoLegal, +read 2026-10-30) permits reproducing its content without profit as long +as the source is cited, which the credit line does. The signatories are +named public officials, but they are real people, so the corpus stays in +the cache, like every local corpus. +""" +from __future__ import annotations + +import datetime +import gzip +import html.parser +import json +import re +import sys +import zlib +from collections import Counter +from collections.abc import Iterator +from pathlib import Path +from typing import Any + +import corpus_build +from local_core import (Fetched, _pinned, _refuse_past, _retried, directory, + fetch_command) + + +DOF = "dof-signatures" +SIDOF = "https://sidof.segob.gob.mx" +DOF_DAY_URL = SIDOF + "/dof/sidof/notas/{day}" +DOF_NOTE_URL = SIDOF + "/dof/sidof/notas/nota/{cod}" +DOF_ROBOTS_URL = SIDOF + "/robots.txt" +#: The days read, both included: the whole of 2025, the last full year +#: on 2026-10-30. +DOF_FROM = datetime.date(2025, 1, 1) +DOF_TO = datetime.date(2025, 12, 31) +#: Seconds slept after each request. +DOF_DELAY = 0.25 +#: The day listing's three editions. +DOF_EDITIONS = ("NotasMatutinas", "NotasVespertinas", "NotasExtraordinarias") +#: How a note's HTML is read into excerpts (EXCERPT_BLOCKS, the block +#: tags, what counts as bold), stored in each day file and in meta.json: +#: a change to excerpts() must bump it, and a day cached under another +#: is fetched again. +DOF_EXCERPT_VERSION = 1 +#: How excerpts are cut into records (cut(), signatures()), in +#: meta.json's detail: a change must bump it, and a corpus cut the old +#: way is refused on load (check_dof). No refetch is needed: the cut is +#: read from the cached days. +DOF_CUT_VERSION = 4 +#: The blocks kept before each block holding a "Rúbrica": the stacked +#: signature's office lines and name, or a date line before them. +EXCERPT_BLOCKS = 3 +#: Gates: listed notes with HTML whose note came back without it (or +#: flagged too large to send), and "Rúbrica"s no rule could cut. +DOF_MAX_EMPTY = 0.05 +DOF_MAX_UNCUT = 0.10 + +_DAYS = "days" + + +# -- reading a note's HTML --------------------------------------------------- + +_BLOCK_TAGS = frozenset({"div", "p", "td", "th", "tr", "br", "li", "table", + "h1", "h2", "h3", "h4", "h5", "h6"}) +_SKIPPED_TAGS = frozenset({"style", "script", "title", "head"}) +_VOID_TAGS = frozenset({"br", "img", "hr", "meta", "link", "input", "col"}) +_BOLD_STYLE = re.compile(r"font-weight:\s*(?:bold|[6-9]00)", re.I) + + +class _Blocks(html.parser.HTMLParser): + """A note's text, cut into blocks at the block tags, each character + marked with whether an element around it is bold: a or , + or a bold font-weight. The DOF sets a signatory's name in bold, + which is what cut() reads first.""" + + def __init__(self) -> None: + super().__init__(convert_charrefs=True) + self.blocks: list[list[tuple[str, bool]]] = [[]] + self._open: list[tuple[str, bool]] = [] + self._skipping = 0 + + def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None: + if tag in _SKIPPED_TAGS: + self._skipping += 1 + return + if tag in _BLOCK_TAGS: + self.blocks.append([]) + if tag in _VOID_TAGS: + return + style = next((v or "" for k, v in attrs if k == "style"), "") + self._open.append((tag, tag in ("b", "strong") or bool(_BOLD_STYLE.search(style)))) + + def handle_endtag(self, tag: str) -> None: + if tag in _SKIPPED_TAGS: + self._skipping = max(0, self._skipping - 1) + return + for at in range(len(self._open) - 1, -1, -1): + if self._open[at][0] == tag: + del self._open[at:] + break + if tag in _BLOCK_TAGS: + self.blocks.append([]) + + def handle_data(self, data: str) -> None: + if not self._skipping: + self.blocks[-1].append((data, any(bold for _, bold in self._open))) + + +def _block(runs: list[tuple[str, bool]]) -> tuple[str, str]: + """A block's runs as its text, whitespace read as single spaces and + trimmed, and a mask of the same length, "1" under a bold character.""" + text: list[str] = [] + mask: list[str] = [] + for data, bold in runs: + for char in data: + if char.isspace(): + if text and text[-1] != " ": + text.append(" ") + mask.append("0") + else: + text.append(char) + mask.append("1" if bold else "0") + while text and text[-1] == " ": + text.pop() + mask.pop() + return "".join(text), "".join(mask) + + +RUBRICA = re.compile(r"r[úu]brica", re.I) + + +def excerpts(content: str) -> list[list[tuple[str, str]]]: + """The excerpts of a note's HTML that cut() reads: for each block + holding a "Rúbrica", that block with the EXCERPT_BLOCKS before it, + each a (text, bold mask) pair. Runs of such blocks share one + excerpt, so a signature's blocks are never split between two.""" + parser = _Blocks() + parser.feed(content) + parser.close() + blocks = [b for b in (_block(runs) for runs in parser.blocks) if b[0]] + found: list[list[tuple[str, str]]] = [] + start = -1 + for at, (text, _) in enumerate(blocks): + if not RUBRICA.search(text): + continue + if found and at - EXCERPT_BLOCKS <= start + 1: + found[-1].extend(blocks[start + 1:at + 1]) + else: + found.append(blocks[max(0, at - EXCERPT_BLOCKS):at + 1]) + start = at + return found + + +# -- cutting a name ---------------------------------------------------------- + +#: The spelled-out titles a name may open with, as the DOF writes them, +#: matched case-insensitively. An abbreviated title is told by its shape +#: instead (_ABBREVIATION). +SPELLED_TITLES = frozenset({ + "licenciado", "licenciada", "maestro", "maestra", "doctor", "doctora", + "ingeniero", "ingeniera", "arquitecto", "arquitecta", "contador", + "contadora", "ciudadano", "ciudadana", "profesor", "profesora", + "actuario", "actuaria", "notario", "notaria"}) +#: An abbreviated title: one or more chunks of a capital and up to four +#: lower-case letters, each ending in a full stop (`Lic.`, `C.`, `C.P.`, +#: `L.C.P.`, `Mtra.`, `Lcda.`), or capitals alone (`LIC.`). +_ABBREVIATION = re.compile(r"(?:[A-ZÁÉÍÓÚÑ](?:[a-záéíóúñ]{0,4}|[A-ZÁÉÍÓÚÑ]{0,4})\.)+") +#: An article before a title or a name (`la Lic. ...`, `el C. ...`), +#: left out of the cut. +_ARTICLES = frozenset({"el", "la", "los", "las"}) +# What a cut is refused as, though a rule cut it: a date (`a siete de +# noviembre de dos mil veinticinco`, `30 DE MAYO DE 2025`), and an office +# signing without a name (`EL +# SECRETARIO DE SALUD`), which opens with an article where a name does +# not. A bare digit or month is not enough: `CAP. 1/O. AYTE.` is a rank +# and `Julio` a given name; `Del Olmo` is a surname, so `del` is not an +# article here. +_MONTHS = ("enero|febrero|marzo|abril|mayo|junio|julio|agosto|septiembre|setiembre" + "|octubre|noviembre|diciembre") +_DATE = re.compile(rf"\bde (?:{_MONTHS})\b|\bdos mil\b|\b\d{{4}}\b", re.I) +#: The end of the sentence before a signature: `.-` or `.–`. +_CLAUSE_START = re.compile(r"\.\s*[-–—]\s*") +#: What ends the text before a "Rúbrica" in a signature: a dash, or a +#: full stop, with the spaces around it. +_BEFORE_RUBRICA = re.compile(r"(?:\.\s*)?[-–—]?\s*$") +_EDGE = " ,;:.-–—\"'()" +#: The most words a title rule's cut may hold: the longest names in 400 +#: cuts checked by hand ran to 7 (`CAP. 1/O. AYTE. CONTAB. DANIEL LARA +#: HERNANDEZ`). +_TITLE_WORDS = 8 + + +def is_title(word: str) -> bool: + """Whether `word` is a title a name may open with, by cut()'s test.""" + return (bool(_ABBREVIATION.fullmatch(word)) + or word.lower().strip(".") in SPELLED_TITLES) + + +def _words(text: str, mask: str) -> list[tuple[int, int, bool]]: + """`text`'s words as (start, end, bold): bold when any letter in + the word is.""" + out = [] + for match in re.finditer(r"\S+", text): + bold = any(mask[i] == "1" and text[i].isalpha() + for i in range(match.start(), match.end())) + out.append((match.start(), match.end(), bold)) + return out + + +def _with_titles(text: str, words: list[tuple[int, int, bool]], first: int) -> int: + """The index of the first word of the run of titles directly before + word `first`, or `first` when none is.""" + while first > 0 and is_title(text[words[first - 1][0]:words[first - 1][1]]): + first -= 1 + return first + + +def _span(text: str, words: list[tuple[int, int, bool]], first: int, last: int) -> str: + return text[words[first][0]:words[last][1]].strip(_EDGE + " ").rstrip(_EDGE) + + +def cut(text: str, mask: str) -> tuple[str, str] | None: + """The name in a signature's clause and the rule that cut it, or + None when no rule finds one. The rules, in order: + + - "bold": the last run of words with a bold letter, the run ending + at a word that ends in a comma (`Pardo, Presidenta ...` keeps + `Pardo`), with any titles directly before it (`Lic.`, which the + DOF mostly leaves in roman); + - "title": no bold, so the run from the last title (is_title) with + a word behind it, to the first comma after it (`Sen. Ana Ruiz, + Secretaria` keeps `Sen. Ana Ruiz`), and no longer than + _TITLE_WORDS words, past which it is a sentence a title opened. + + Nothing else cuts: a clause with neither is mostly an office, a + motto or a sentence, and in 2025 what follows its last comma was a + name in 2 of 46 distinct cuts, the whole clause in about 5 of 60 + (measured 2026-10-31, cut version 1). + """ + words = _words(text, mask) + if not words: + return None + bold = [at for at, (_, _, b) in enumerate(words) if b] + if bold: + last = bold[-1] + first = last + while (first > 0 and words[first - 1][2] + and not text[words[first - 1][0]:words[first - 1][1]].endswith((",", ";", ":"))): + first -= 1 + name = _span(text, words, _with_titles(text, words, first), last) + return (name, "bold") if name else None + titles = [at for at in range(len(words) - 1) + if is_title(text[words[at][0]:words[at][1]])] + if not titles: + return None + first = _with_titles(text, words, titles[-1]) + last = next((at for at in range(titles[-1] + 1, len(words)) + if text[words[at][0]:words[at][1]].endswith(",")), len(words) - 1) + if last - first + 1 > _TITLE_WORDS: + return None + name = _span(text, words, first, last) + return (name, "title") if name else None + + +def joined(before: str) -> str: + """How a cut name meets the clause before it: "start" (nothing), "comma", + "colon", "article" (`la`, `el`), "block" (a stacked signature, the + office in the block above), or "word": glued to a word with no + punctuation between, the shape a title marks the start of a name in. + `before` is the clause's text before the cut, blocks joined by " | ".""" + rest = before.rstrip() + if not rest: + return "start" + if rest.endswith("|"): + return "block" + if rest.endswith(","): + return "comma" + if rest.endswith((":", ";")): + return "colon" + if rest.split()[-1].lower() in _ARTICLES: + return "article" + return "word" + + +def signatures(excerpt: list[tuple[str, str]]) -> Iterator[dict[str, str] | str]: + """Each signature in an excerpt, as a record without its source: the + name cut, the rule, the title the name opens with ("" for none), how + it meets the clause (joined()), the layout and the raw clause; or, + for a "Rúbrica" that cut() found no name before, or whose cut is one + word, a date or an office (_DATE, _ARTICLES), the reason, as a string, which + the fetch counts. + + Two layouts. "stacked": a block that is the "Rúbrica" alone, the + name the block above it and the office the blocks above that, which + the clause holds joined by " | ". "inline": `Office, Title Name.- + Rúbrica.-` in a paragraph, the clause running from the last `.-` + before the name, so it may hold the end of an attestation (`Doy + fe`, `Atentamente`).""" + for at, (text, mask) in enumerate(excerpt): + for match in RUBRICA.finditer(text): + before = text[:match.start()] + if not before.strip(_EDGE + " *|"): + if at == 0 or RUBRICA.search(excerpt[at - 1][0]): + yield "stacked, no block above" + continue + name_text, name_mask = excerpt[at - 1] + above = [t for t, _ in excerpt[max(0, at - 1 - (EXCERPT_BLOCKS - 1)):at - 1] + if not RUBRICA.search(t)] + layout = "stacked" + prefix = "".join(f"{t} | " for t in above) + else: + end = _BEFORE_RUBRICA.search(before) + body = before[:end.start()] if end else before + if not body.strip() or body.rstrip()[-1:] in ",;:": + yield "inline, nothing before" + continue + starts = list(_CLAUSE_START.finditer(body)) + begin = starts[-1].end() if starts else 0 + name_text, name_mask = body[begin:], mask[begin:len(body)] + layout, prefix = "inline", "" + found = cut(name_text, name_mask) + if found is None: + yield f"{layout}, no rule cut a name" + continue + name, rule = found + if len(name.split()) == 1: # `f`, `III`, `Atentamente`, a surname cut off + yield f"{layout}, cut one word" + continue + if _DATE.search(name): + yield f"{layout}, cut a date" + continue + if name.split()[0].lower() in _ARTICLES: + yield f"{layout}, cut an office" + continue + clause = prefix + name_text + where = clause.rfind(name) + first = name.split()[0] + yield {"name": name, "cut": rule, "layout": layout, + "title": first if is_title(first) else "", + "joined": joined(clause[:where] if where >= 0 else clause), + "clause": clause} + + +# -- fetching ---------------------------------------------------------------- + +def _days() -> Iterator[datetime.date]: + day = DOF_FROM + while day <= DOF_TO: + yield day + day += datetime.timedelta(days=1) + + +def _day_path(day: datetime.date) -> Path: + return directory(DOF) / _DAYS / f"{day.isoformat()}.json.gz" + + +def _get(url: str) -> dict[str, Any]: + """`url`'s JSON object, required to answer messageCode 200.""" + data = _retried(url, lambda body: corpus_build.parse_json(body, url), DOF_DELAY, + timeout=120) + if not isinstance(data, dict) or data.get("messageCode") != 200: + code = data.get("messageCode") if isinstance(data, dict) else type(data).__name__ + raise ValueError(f"{url} answered {code}, not messageCode 200") + return data + + +def disallowed(robots: str) -> frozenset[int]: + """The notes robots.txt disallows for every agent: its + `Disallow: /notas/` lines under `User-agent: *`.""" + found: set[int] = set() + every = False + for line in robots.splitlines(): + key, _, value = line.partition(":") + key, value = key.strip().lower(), value.strip() + if key == "user-agent": + every = value == "*" + elif every and key == "disallow": + if match := re.fullmatch(r"/notas/(?:docFuente/)?(\d+)", value): + found.add(int(match.group(1))) + return frozenset(found) + + +def _fetch_day(day: datetime.date, skip: frozenset[int]) -> dict[str, Any]: + """A day's listing and every listed note with HTML, read into + excerpts; the counts read past are kept with them.""" + listing = _get(DOF_DAY_URL.format(day=day.strftime("%d-%m-%Y"))) + notes: list[dict[str, Any]] = [] + counts: Counter[str] = Counter() + for edition in DOF_EDITIONS: + for entry in listing.get(edition) or []: + cod = int(entry["codNota"]) + if entry.get("existeHtml") != "S": + counts["without html"] += 1 + continue + if cod in skip: + counts["disallowed"] += 1 + continue + note = _get(DOF_NOTE_URL.format(cod=cod))["Nota"] + content = note.get("cadenaContenido") or "" + if not content: + # A note published as page images only is listed with + # HTML and states its size as 0 (2,936 of the 26,752 + # 2025 listed with HTML, spread over every month); a + # blank note of stated size (22) is what the gate is for. + image_only = str(note.get("tamanioContenido")) == "0" + counts["image only" if image_only else "empty"] += 1 + continue + counts["read"] += 1 + notes.append({"note": cod, "edition": edition, + "excerpts": excerpts(content)}) + return {"day": day.isoformat(), "excerpt_version": DOF_EXCERPT_VERSION, + "counts": dict(counts), "notes": notes} + + +def _read_day(path: Path) -> dict[str, Any] | None: + """A cached day, or None where none is cached or it was read under + another DOF_EXCERPT_VERSION. A file that is not a whole gzip of one + JSON object raises, naming the way out.""" + if not path.exists(): + return None + try: + day = json.loads(gzip.decompress(path.read_bytes())) + except (EOFError, gzip.BadGzipFile, zlib.error, ValueError) as exc: + raise ValueError(f"{path} is damaged ({type(exc).__name__}: {exc}); delete " + f"it and fetch again: {fetch_command(DOF)}") from None + if not isinstance(day, dict) or day.get("excerpt_version") != DOF_EXCERPT_VERSION: + return None + return day + + +def _cache_days(*, again: bool = False) -> list[dict[str, Any]]: + """Every day of DOF_FROM..DOF_TO, from the cache where it holds the + day (unless `again`), else fetched and cached.""" + skip = disallowed(_retried(DOF_ROBOTS_URL, lambda b: b.decode("utf-8"), DOF_DELAY)) + (directory(DOF) / _DAYS).mkdir(exist_ok=True) + out = [] + for number, day in enumerate(_days(), 1): + path = _day_path(day) + held = None if again else _read_day(path) + if held is None: + held = _fetch_day(day, skip) + body = (corpus_build.jsonl_line(held, sort_keys=True) + "\n").encode() + corpus_build.write_atomic(path, gzip.compress(body, mtime=0)) + print(f" {day.isoformat()}: {held['counts'].get('read', 0)} notes " + f"({number} of {(DOF_TO - DOF_FROM).days + 1} days)", file=sys.stderr) + out.append(held) + return out + + +def fetch_dof() -> Fetched: + """One record per signature, from the days cached (fetching any not + cached): the cut name, its rule, title and join (signatures()), the + clause, the day and the note. + + A listed note that states a size but comes back without HTML (one + stating 0 is image only, and read past), or a "Rúbrica" no rule + cut, is counted, and past DOF_MAX_EMPTY or DOF_MAX_UNCUT of + the notes or the "Rúbrica"s the fetch is refused: a changed layout + would otherwise read as a year with fewer signatures.""" + days = _cache_days() + records: list[dict[str, str]] = [] + counts: Counter[str] = Counter() + uncut: Counter[str] = Counter() + for day in days: + counts.update(day["counts"]) + for note in day["notes"]: + for excerpt in note["excerpts"]: + for found in signatures([tuple(block) for block in excerpt]): + if isinstance(found, str): + uncut[found] += 1 + continue + records.append({**found, "day": day["day"], "note": str(note["note"])}) + listed = counts["read"] + counts["empty"] + marks = len(records) + sum(uncut.values()) + _refuse_past([(counts["empty"], listed, DOF_MAX_EMPTY, "notes with HTML came back empty"), + (sum(uncut.values()), marks, DOF_MAX_UNCUT, "Rúbricas cut no name")], + "the DOF's layout may have changed under the reader") + return Fetched(records, DOF_TO.isoformat(), { + "from": DOF_FROM.isoformat(), "to": DOF_TO.isoformat(), + "excerpt_version": DOF_EXCERPT_VERSION, "cut_version": DOF_CUT_VERSION, + "notes": dict(counts), "uncut": dict(uncut), + "cuts": dict(Counter(r["cut"] for r in records))}) + + +def renew_dof() -> None: + """What `fetch --refresh` runs first: every day fetched again, each + replacing its cached file once read whole. The DOF corrects a note + by publishing another, so a day once cached seldom changes.""" + _cache_days(again=True) + + +_check_range = _pinned("from", lambda: DOF_FROM.isoformat(), + "was fetched from {held}, and this tool reads from {wanted}") +_check_to = _pinned("to", lambda: DOF_TO.isoformat(), + "was fetched to {held}, and this tool reads to {wanted}") +_check_cut = _pinned("cut_version", lambda: DOF_CUT_VERSION, + "was cut by cut version {held}, and this tool cuts by " + "version {wanted}; fetch again, which reads the cached days") + + +def check_dof(meta: dict[str, Any]) -> str: + """Why a corpus of other days or another cut is refused.""" + return _check_range(meta) or _check_to(meta) or _check_cut(meta) From d7b41eebe768f9b9f24a7c124d7df8a19acab8f3 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 31 Oct 2026 04:32:27 -0700 Subject: [PATCH 2/3] Harden the dof-signatures fetch and cut after review - check_dof pins excerpt_version beside cut_version, so a corpus cut from excerpts of another version is refused on load. - A day listing lacking any of the three editions, or holding one that is not a list, is refused rather than cached short; and a gate refuses past 5% of weekdays listing no note (7 of 261 in 2025, each a public holiday; every 2025 listing read again held all three editions and as many notes as cached). - Image-only notes (stated size 0, 11.0% in 2025) are gated at 20%. - cut()'s docstring now says what the bold rule does: of a run in bold throughout, the part after its last comma is kept (the name in about 78 of 90 such signatures). A cut wholly in capitals opening with an office's head word or with DE/DEL is refused as an office (52 signatures; one name lost, CONSEJERO ...). - robots.txt is read only once a day needs fetching, so a re-cut asks nothing of the site, as the docstring said. - A mixed-case abbreviation is a title only from a list, so `Ma.`, `Gpe.`, `Eds.` and a surname ending a sentence (`Paz. Conste`) no longer are; capitals keep the shape. - A messageCode other than 200 is retried like a body cut short. - The hand-check counts are dated and keyed to their cut versions. Cut version 5: 33,564 signatures, 6,643 distinct names; of 200 cuts checked by hand, 199 exactly a name and 1 a name behind a rank. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- tests/v2/test_local_dof.py | 96 ++++++++++++++++---- tools/differential/README.md | 2 +- tools/differential/local_dof.py | 151 ++++++++++++++++++++++++-------- 4 files changed, 196 insertions(+), 55 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 0f6c07e3..0344346e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -256,7 +256,7 @@ The 2.0 rewrite landed as underscore-private modules beside the v1 code, which 2 **Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `AFTER_GIVEN_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `nameparser/_reading_index.py`'s `BEHAVIORAL` table, a branch in its `compile_readings` adding the words whose reading carries it and a key in its `Compiled` TypedDict, a constant of the same name in `tools/vocab/build.py`'s `CONSTANTS` roster (an import-time assert there holds the two to one list), and the `NAME = _compiled.NAME` assignment in the config module. A property the compiler reads is kept in the reading index by `indexed()`, which reads `BEHAVIORAL`, so a new one reaches a choice of languages too. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A marker that records a clash with a NAME is the other shape: derived from the word's name reading beside the clashing one (`SUFFIX_ACRONYMS_AMBIGUOUS`, `SUFFIX_WORDS_AMBIGUOUS`, and the complement `NON_GIVEN_NAME_PARTICLES` works from), not written as a property, so it too holds by construction (decisions/vocabulary-model, 2026-10-25). A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). A caller's `Lexicon.add()` to a sub-set does **not** add the word to its base, so a caller adding a brand-new word adds it to both. Two traps a gating set met (#752 phase 2i). Where a caller listing a word in the BASE set has always meant "read it as this everywhere" (the 1.4 recipe `c.titles.add("dean")`), the gated words compile OUT of the base rather than into both, or the gate overrides the caller's listing: `TITLES_AMBIGUOUS` is not a subset of `TITLES` for that reason, and a word in both reads as a title. And `_VOCAB_FIELDS` is read whole by two rosters in `_pipeline/_vocab.py`, the capitals shape's "unlisted" (`in_any_wordlist`) and the comma contrast's "claimed as non-name" (`_NON_NAME_FIELDS`), so a new field joins both by default; a field whose words may be names leaves both, which the grid found as `John Smith, DEAN` reversed where C6 had read a credential; #789's `given_name_abbreviations` met it again (`John Smith, JAS` and `Jas SMITH, XYZ` reversed), found by review because its grid held only the dotted spellings the set was for -- a new field's grid holds the spellings it must NOT move too. -**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials --base `, which reads the base from a baseline kept for its package's files and never the shared snapshot beside the corpus (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating; the shared snapshot beside the corpus is one file per source for every worktree, and #752's phase 2f found ctgov reporting a sibling branch's 1,334 joined-credential readings as its own moves against it); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to English titles, honorifics or post-nominals also reads `snapshot.py --local govuk-people` the same way: GOV.UK's display names of ministers, officials and officers, the title run and the post-nominals as an office writes them (`The Rt Hon`, `Sir`, `Baroness`, ranks; `KC`, `CBE`, `KCB`), in 2,093 of whose 6,021 `parse()` read one or both on 2026-10-30. A change to Spanish titles, to the `es` pack, or to how a title marks where a name starts after an office reads `snapshot.py --local dof-signatures` the same way: the names in the signature blocks of Mexico's Diario Oficial for 2025, cut from the clause that names the signatory's office, 65.3% of its 33,623 signatures opening with a title (`Lic.`, `Dr.`, `Mtro.`, `C.`, `Licenciada`, `C.P.`), whose first word `parse()` read as a title in 82.1% and `parser_for(locales.ES)` in 86.8% on 2026-10-31. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific titles such as `Guru` in front, `Shekh`, a title there until #752's phase 2j made it one only inside a title phrase, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals --base ` the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality; and `local_agreement.py orcid-researchers --base `, ORCID holders' own two boxes (given names, family name) from every country, scored by the country they work or live in -- a population each person split for themselves, academic but worldwide, the one to read for a change a language pack makes; and `local_agreement.py patentsview-inventors --base `, US patents' inventors in the two boxes their filings give, split by a professional filer, with how many people filed under each name in each country, the one to read for how common a name's word is among a country's US-patent inventors. A change to titles or post-nominals, or to the `cs` pack, reads `local_agreement.py cz-candidates --base `, and with `--locale cs` too: Czech election candidates' four boxes (the title before the name, given names, surname, the title after it), scored by election, the only population whose titles are stated apart from the name, 36% of its 170,181 records titled (2026-10-30), nearly all academic degrees; its names are composed from the boxes (`Ing. Jan Novák, Ph.D.`), so it says nothing about how a writer spaces or punctuates them. These are the only real fielded populations outside the committed fields files, and no CI run sees them either. +**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials --base `, which reads the base from a baseline kept for its package's files and never the shared snapshot beside the corpus (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating; the shared snapshot beside the corpus is one file per source for every worktree, and #752's phase 2f found ctgov reporting a sibling branch's 1,334 joined-credential readings as its own moves against it); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to English titles, honorifics or post-nominals also reads `snapshot.py --local govuk-people` the same way: GOV.UK's display names of ministers, officials and officers, the title run and the post-nominals as an office writes them (`The Rt Hon`, `Sir`, `Baroness`, ranks; `KC`, `CBE`, `KCB`), in 2,093 of whose 6,021 `parse()` read one or both on 2026-10-30. A change to Spanish titles, to the `es` pack, or to how a title marks where a name starts after an office reads `snapshot.py --local dof-signatures` the same way: the names in the signature blocks of Mexico's Diario Oficial for 2025, cut from the clause that names the signatory's office, 65.4% of its 33,564 signatures opening with a title (`Lic.`, `Dr.`, `Mtro.`, `C.`, `Licenciada`, `C.P.`), whose first word `parse()` read as a title in 82.1% and `parser_for(locales.ES)` in 86.8% on 2026-10-31. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific titles such as `Guru` in front, `Shekh`, a title there until #752's phase 2j made it one only inside a title phrase, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals --base ` the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality; and `local_agreement.py orcid-researchers --base `, ORCID holders' own two boxes (given names, family name) from every country, scored by the country they work or live in -- a population each person split for themselves, academic but worldwide, the one to read for a change a language pack makes; and `local_agreement.py patentsview-inventors --base `, US patents' inventors in the two boxes their filings give, split by a professional filer, with how many people filed under each name in each country, the one to read for how common a name's word is among a country's US-patent inventors. A change to titles or post-nominals, or to the `cs` pack, reads `local_agreement.py cz-candidates --base `, and with `--locale cs` too: Czech election candidates' four boxes (the title before the name, given names, surname, the title after it), scored by election, the only population whose titles are stated apart from the name, 36% of its 170,181 records titled (2026-10-30), nearly all academic degrees; its names are composed from the boxes (`Ing. Jan Novák, Ph.D.`), so it says nothing about how a writer spaces or punctuates them. These are the only real fielded populations outside the committed fields files, and no CI run sees them either. ## Gotchas diff --git a/tests/v2/test_local_dof.py b/tests/v2/test_local_dof.py index 98c7eed5..eab3dea7 100644 --- a/tests/v2/test_local_dof.py +++ b/tests/v2/test_local_dof.py @@ -90,6 +90,9 @@ def test_a_title_written_in_capitals_is_read_as_one() -> None: ("Directora de Área, la C.P. Ana Lis Mora", ("C.P. Ana Lis Mora", "title")), # The office after the name, set off by a comma, stays out. ("Sen. Ana Lis Mora, Secretaria", ("Sen. Ana Lis Mora", "title")), + # A surname ending a sentence is not a title: the run opens at `Lic.`, + # not at `Paz.`, though it still runs on to the end of the clause. + ("firmado por Lic. Ana Ruiz Paz. Conste", ("Lic. Ana Ruiz Paz. Conste", "title")), ]) def test_without_bold_a_title_opens_the_cut(clause: str, expected: tuple[str, str]) -> None: assert dof.cut(clause, "0" * len(clause)) == expected @@ -121,12 +124,13 @@ def test_how_a_cut_meets_the_clause_before_it(before: str, joined: str) -> None: @pytest.mark.parametrize("word", ["Lic.", "C.", "C.P.", "L.C.P.", "Mtro.", "Lcda.", "LIC.", - "Licenciado", "MAESTRA"]) + "MVZ.", "Gral.", "Biól.", "Licenciado", "MAESTRA"]) def test_titles_by_shape_and_by_spelling(word: str) -> None: assert dof.is_title(word) -@pytest.mark.parametrize("word", ["Ana", "Rúbrica.", "de", "Lic", "CONSEJO."]) +@pytest.mark.parametrize("word", ["Ana", "Rúbrica.", "de", "Lic", "CONSEJO.", + "Paz.", "Ma.", "Gpe.", "Eds."]) def test_words_that_are_not_titles(word: str) -> None: assert not dof.is_title(word) @@ -148,6 +152,31 @@ def test_a_cut_of_one_word_a_date_or_an_office_is_counted_not_kept(name: str, re assert _records(signed) == [f"stacked, {reason}"] +def test_a_run_in_bold_throughout_keeps_the_part_after_its_last_comma() -> None: + rank = "GRAL. BRIG. E.M., Ana Sol Pardo" + assert dof.cut(rank, "1" * len(rank)) == ("Ana Sol Pardo", "bold") + office = "Ana Sol Pardo, Presidenta de la Comisión" + assert dof.cut(office, "1" * len(office)) == ("Presidenta de la Comisión", "bold") + # an office after the name in roman stays out of the run + roman = "Ana Sol Pardo, Presidenta" + assert dof.cut(roman, "1" * len("Ana Sol Pardo,") + "0" * len(" Presidenta")) == ( + "Ana Sol Pardo", "bold") + + +@pytest.mark.parametrize(("name", "office"), [ + ("DIRECTORA GENERAL DE RECURSOS HUMANOS", True), + ("C. GOBERNADOR CONSTITUCIONAL DEL ESTADO", True), + ("DEL ESTADO DE YUCATÁN", True), + ("EL SECRETARIO DE SALUD", True), + ("Magistrado José Rey Paz", False), # an office written as a title, before a name + ("Jueza de Distrito Ana Lis Mora", False), + ("Del Olmo Blanco", False), # a surname + ("LIC. ANA LIS MORA", False), +]) +def test_an_office_signing_without_a_name(name: str, office: bool) -> None: + assert dof._office(name) is office + + @pytest.mark.parametrize("name", [ "Lic. Julio Linares Rangel", "Del Olmo Blanco", "CAP. 1/O. AYTE. LUIS REY PAZ"]) def test_a_month_a_digit_or_del_inside_a_name_is_kept(name: str) -> None: @@ -172,7 +201,7 @@ def test_robots_disallows_only_the_notes_listed_for_every_agent() -> None: # -- the fetch ----------------------------------------------------------------- -_DAY1, _DAY2 = datetime.date(2025, 1, 2), datetime.date(2025, 1, 3) +_DAY1, _DAY2 = datetime.date(2025, 1, 3), datetime.date(2025, 1, 4) def _listing(*entries: dict[str, Any]) -> dict[str, Any]: @@ -185,10 +214,10 @@ def _entry(cod: int, html: str = "S") -> dict[str, Any]: #: Day 1: the stacked note, the inline note, a section heading without -#: HTML and a note robots.txt disallows. Day 2: no edition. +#: HTML and a note robots.txt disallows. Day 2, a Saturday: no note. _NOTES: dict[int, str | None] = {1: _STACKED, 2: _INLINE, 9: _CAPITALS} -_DAYS = {"02-01-2025": _listing(_entry(1), _entry(2), _entry(3, "N"), _entry(9)), - "03-01-2025": _listing()} +_DAYS = {"03-01-2025": _listing(_entry(1), _entry(2), _entry(3, "N"), _entry(9)), + "04-01-2025": _listing()} def _serve(monkeypatch: pytest.MonkeyPatch, *, @@ -230,23 +259,43 @@ def test_a_dof_fetch_walks_the_days_and_keeps_each_signature( asked = _serve(monkeypatch) corpus = local.fetch(_DOF) assert [(r["name"], r["day"], r["note"]) for r in corpus.records] == [ - ("Lic. Ana Ruiz Toledo", "2025-01-02", "1"), ("Mtra. Eva Mar Soto", "2025-01-02", "2"), - ("Bo Gil Paz", "2025-01-02", "2")] + ("Lic. Ana Ruiz Toledo", "2025-01-03", "1"), ("Mtra. Eva Mar Soto", "2025-01-03", "2"), + ("Bo Gil Paz", "2025-01-03", "2")] assert dof.DOF_NOTE_URL.format(cod=9) not in asked - assert corpus.meta["data_date"] == "2025-01-03" + assert corpus.meta["data_date"] == "2025-01-04" detail = corpus.meta["detail"] assert detail["notes"] == {"read": 2, "without html": 1, "disallowed": 1} assert (detail["cuts"], detail["uncut"]) == ({"bold": 3}, {}) assert corpus.header().endswith("3 signatures (local corpus dof-signatures)") -def test_a_second_fetch_reads_the_cached_days_and_asks_only_robots( +def test_a_second_fetch_reads_the_cached_days_and_asks_nothing( monkeypatch: pytest.MonkeyPatch) -> None: _serve(monkeypatch) first = local.fetch(_DOF).records asked = _serve(monkeypatch) assert local.fetch(_DOF).records == first - assert asked == [dof.DOF_ROBOTS_URL] + assert asked == [] # robots.txt is read only once a day needs fetching + + +def test_a_listing_without_one_of_the_editions_is_refused( + monkeypatch: pytest.MonkeyPatch) -> None: + for broken in ({k: v for k, v in _listing().items() if k != "NotasVespertinas"}, + {**_listing(), "NotasExtraordinarias": None}): + _serve(monkeypatch, days={**_DAYS, "04-01-2025": broken}) + with pytest.raises(ValueError, match="lists no Notas"): + local.fetch(_DOF) + assert not dof._day_path(_DAY2).exists() + + +def test_weekdays_listing_no_note_past_the_gate_refuse_the_fetch( + monkeypatch: pytest.MonkeyPatch) -> None: + _serve(monkeypatch, days={**_DAYS, "05-01-2025": _listing(), "06-01-2025": _listing()}) + monkeypatch.setattr(dof, "DOF_TO", datetime.date(2025, 1, 6)) # a Sunday, a silent Monday + with pytest.raises(ValueError, match="1 of 2 weekdays listed no note"): + local.fetch(_DOF) + monkeypatch.setattr(dof, "DOF_MAX_SILENT", 0.5) + assert len(local.fetch(_DOF).records) == 3 def test_a_day_cached_under_another_excerpt_version_is_fetched_again( @@ -258,8 +307,8 @@ def test_a_day_cached_under_another_excerpt_version_is_fetched_again( path.write_bytes(gzip.compress(json.dumps({**held, "excerpt_version": 0}).encode())) asked = _serve(monkeypatch) local.fetch(_DOF) - assert dof.DOF_DAY_URL.format(day="02-01-2025") in asked - assert dof.DOF_DAY_URL.format(day="03-01-2025") not in asked + assert dof.DOF_DAY_URL.format(day="03-01-2025") in asked + assert dof.DOF_DAY_URL.format(day="04-01-2025") not in asked def test_a_damaged_cached_day_is_refused_naming_the_way_out( @@ -279,10 +328,15 @@ def test_a_dof_request_failing_once_is_retried(monkeypatch: pytest.MonkeyPatch) assert asked.count(note) == 3 -def test_an_answer_that_is_not_message_code_200_is_refused( +def test_an_answer_that_is_not_message_code_200_is_retried_then_refused( monkeypatch: pytest.MonkeyPatch) -> None: - day = dof.DOF_DAY_URL.format(day="02-01-2025") + day = dof.DOF_DAY_URL.format(day="03-01-2025") + once = [b'{"messageCode": 500}'] + asked = _serve(monkeypatch, fail=lambda url: once.pop() if url == day and once else None) + assert len(local.fetch(_DOF).records) == 3 + assert asked.count(day) == 2 _serve(monkeypatch, fail=lambda url: b'{"messageCode": 500}' if url == day else None) + dof._day_path(_DAY1).unlink() with pytest.raises(ValueError, match="answered 500"): local.fetch(_DOF) @@ -296,9 +350,12 @@ def test_notes_that_came_back_empty_past_the_gate_refuse_the_fetch( assert local.fetch(_DOF).meta["detail"]["notes"]["empty"] == 1 -def test_a_note_published_as_images_is_counted_apart_and_passes_the_gate( +def test_notes_published_as_images_are_counted_apart_and_gated( monkeypatch: pytest.MonkeyPatch) -> None: _serve(monkeypatch, notes={**_NOTES, 2: None}) + with pytest.raises(ValueError, match="1 of 2 notes with HTML stated a size of 0"): + local.fetch(_DOF) + monkeypatch.setattr(dof, "DOF_MAX_IMAGE_ONLY", 0.5) detail = local.fetch(_DOF).meta["detail"] assert detail["notes"] == {"read": 1, "image only": 1, "without html": 1, "disallowed": 1} @@ -318,6 +375,9 @@ def test_a_corpus_cut_by_another_version_is_refused_on_load( monkeypatch.setattr(dof, "DOF_CUT_VERSION", dof.DOF_CUT_VERSION + 1) with pytest.raises(local.CorpusDamaged, match="cut by cut version"): local.load(_DOF) - monkeypatch.setattr(dof, "DOF_TO", datetime.date(2025, 1, 4)) - with pytest.raises(local.CorpusDamaged, match="fetched to 2025-01-03"): + monkeypatch.setattr(dof, "DOF_EXCERPT_VERSION", dof.DOF_EXCERPT_VERSION + 1) + with pytest.raises(local.CorpusDamaged, match="cut from excerpts of version"): + local.load(_DOF) + monkeypatch.setattr(dof, "DOF_TO", datetime.date(2025, 1, 5)) + with pytest.raises(local.CorpusDamaged, match="fetched to 2025-01-04"): local.load(_DOF) diff --git a/tools/differential/README.md b/tools/differential/README.md index f8daeffe..1a85e2a6 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -304,7 +304,7 @@ Any snapshot reading of 5,000 entries or more, committed or local, parses in a w | `jrc-names` | the person entities of the EU Joint Research Centre's JRC-Names, each with the spellings of its name harvested from news text in many languages and scripts: a record is one distinct spelling, with its entity and its script. The file (`entities.gzip`, 14.8 MB) is pinned by sha256 and dated by its Last-Modified, 2024-03-15. Of its 1,317,586 spellings kept, 1,203,634 are Latin, 66,673 Cyrillic, 17,074 Arabic (7,833 persons, 5,347 of them spelled in Latin too), 13,039 Katakana and 8,016 Han, and 218,822 persons have two or more spellings, 25,415 of them in two scripts or more (a spelling whose letters are in two scripts, or none, not counted as one). The population for spelling and transliteration variants of one name, read by the `groups` view, and for Arabic-script names. Licensed under the JRC-Names EULA, personal and non-transferable, with no redistribution or derivative works, so its views' output and its snapshot stay in the cache like every local corpus's, and here the licence, not only the names being real people's, is why | titles and credentials, since the names are harvested from running text (`parse()` read a title in 3,011 of its 1,315,051 names and a suffix in 1,940, 2026-10-24); the ASCII comma, which none holds (1,510 Arabic spellings hold an Arabic comma, U+060C, which the parser reads as one); the Indic scripts (Devanagari 827 spellings, Bengali 419, the others fewer); people the news did not name | | `nepal-candidates` | every candidate in Nepal's local elections of 2079 BS (2022), from the Election Commission's results site (result.election.gov.np), one file per local body: a record per distinct pair of the Commission's romanization (`name`, the one `parse()` reads) and the name in Devanagari as filed (`devanagari`). The corpus and the word index hold both, so a snapshot and a `name_usage.py` lookup read both scripts; `tally`, which counts what follows a comma, reads the romanized name. Of 126,981 pairs fetched 2026-10-24 (152,960 candidates in 753 files, 26 of whose romanizations were a placeholder such as `- - -` and are dropped), the romanized names carry surnames written as initials (`B.K.`, `K.C.`, `G.C.`), honorific words that are titles in front (`Guru`; `Shekh` until #752's phase 2j made it a title only in a title phrase) and `Md.`, Muhammad's abbreviation, which its period makes a title by shape; the Devanagari ones carry vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and the joiners ZWJ and ZWNJ. The population for romanized South Asian names and for Devanagari. The site states no licence, which the credit line says | case and the comma, since the Commission's romanization is one consistent hand; names a person typed themselves; credentials and generational words; the parliament and provincial candidates, whose files carry Devanagari only; names outside Nepal | | `cz-candidates` | every candidate on the registers the Czech Statistical Office (ČSÚ) publishes on volby.gov.cz for the 2022 municipal elections (`kv2022`) and the 2025 Chamber of Deputies election (`ps2025`): a record is a distinct candidate's four name boxes in one election, the title before the name (`TITULPRED`), the given names (`JMENO`), the surname (`PRIJMENI`) and the title after it (`TITULZA`), with the election; 165,736 records of kv2022's 197,506 candidates and 4,445 of ps2025's 4,475 on 2026-10-30. A record's name is COMPOSED, not written by anyone: the boxes joined in the Czech convention, `TITULPRED JMENO PRIJMENI, TITULZA` (`Ing. Jan Novák, Ph.D.`). 61,320 of its 170,181 records carry a title (58,667 before the name, 7,687 after it), nearly all academic degrees (`Ing.`, `Mgr.`, `Bc.`, `MUDr.`; `DiS.`, `Ph.D.`, `MBA`): the population for titles and post-nominals the filer kept apart from the name. Scored by `local_agreement.py` by election, under the default parser and under `--locale cs`. ČSÚ licenses it under CC BY 4.0, asking that derived data be marked as derived, which the composed names are and the credit line says | case, spacing and punctuation of a writer's own, since the string is composed; the family-comma shape; courtesy titles and offices; names outside Czechia; the elections not in `VOLBY_ELECTIONS` (the index lists kv2002 to kv2026 and ps2002 to ps2025) | -| `dof-signatures` | the signatures in the notes of Mexico's Diario Oficial de la Federación (DOF) for 2025, read through SIDOF's web services (a listing per day, each note's HTML; the notes its robots.txt disallows are skipped), each name cut from the clause that signs it (`local_dof.cut`): the last run of bold words, with the titles before it, which the DOF mostly sets in roman, or with no bold the run a title opens, up to a comma. A record is one signature (a signatory signs many notes): the name, the rule that cut it, the layout (`stacked`, the office on lines above the name and `Rúbrica.` on a line below; `inline`, `Office, Title Name.- Rúbrica.-` in a paragraph), the title the name opens with, how it meets the text before it, and the clause. Of the 35,305 `Rúbrica`s in 2025's notes (fetched 2026-10-31: of the 26,752 notes listed with HTML, 2,936 were published as page images, stating a size of 0, and 22 came back blank though they stated one), 33,623 were cut, to 6,663 distinct names, and the 1,682 left are counted by reason in `meta.json` (neither bold nor a title, an office signing without a name, a date, one word). Of the signatures, 65.3% open with a title (`Lic.` 27.7%, `Dr.`/`Dra.` 9.3%, `Mtro.`/`Mtra.` 8.1%, `C.` 5.1%, `Ing.` 3.6%, a spelled-out `Licenciado`/`Licenciada` 2.9%, a dotted acronym degree such as `C.P.` or `L.C.P.` in most of the rest), and 45.4% stand under their office in a stacked block, glued to it with no comma once the lines are read as one. Of 200 cuts drawn at random and checked by hand on 2026-10-31 (cut version 3), 196 were exactly a name, 3 a name with an office or degree (`Juez`, `Magistrado`, `Doctora en Derecho.`) and 1 not a name (`f`, a one-word cut refused since). The population for Spanish titles in front of a name. The DOF's texts are not protected by copyright (LFDA art. 14 VIII), and SIDOF permits reproduction without profit with the source cited, which the credit line says | the office before the name, which the cut drops, so a snapshot reads names with nothing before them, and the comma; names a person typed themselves, since the DOF's typesetters write every one; private persons, a signatory being an official, a judge or a party's representative; names outside Mexico and in any script but Latin. The `title` field reads a title by shape, so a lone initial counts as one (`J. Guadalupe`, José's) | +| `dof-signatures` | the signatures in the notes of Mexico's Diario Oficial de la Federación (DOF) for 2025, read through SIDOF's web services (a listing per day, each note's HTML; the notes its robots.txt disallows are skipped), each name cut from the clause that signs it (`local_dof.cut`): the last run of bold words, with the titles before it, which the DOF mostly sets in roman, or with no bold the run a title opens, up to a comma. A record is one signature (a signatory signs many notes): the name, the rule that cut it, the layout (`stacked`, the office on lines above the name and `Rúbrica.` on a line below; `inline`, `Office, Title Name.- Rúbrica.-` in a paragraph), the title the name opens with, how it meets the text before it, and the clause. Of the 35,305 `Rúbrica`s in 2025's notes (fetched 2026-10-31: of the 26,752 notes listed with HTML, 2,936 were published as page images, stating a size of 0, and 22 came back blank though they stated one), 33,564 were cut, to 6,643 distinct names, and the 1,741 left are counted by reason in `meta.json` (neither bold nor a title, an office signing without a name, a date, one word); every day's listing was read again on 2026-10-31 and held all three editions and as many notes as cached, the 7 weekdays listing none being public holidays. Of the signatures, 65.4% open with a title (`Lic.` 27.7%, `Dr.`/`Dra.` 9.3%, `Mtro.`/`Mtra.` 8.2%, `C.` 5.1%, `Ing.` 3.6%, a spelled-out `Licenciado`/`Licenciada` 2.9%, a dotted acronym degree such as `C.P.` or `L.C.P.` in most of the rest), and 45.3% stand under their office in a stacked block, glued to it with no comma once the lines are read as one. Of 200 cuts drawn at random and checked by hand on 2026-10-31 under cut version 5, the one fetched, 199 were exactly a name and 1 a name behind a rank (`VICEALMIRANTE`), none not a name; under cut version 3, 196, 3 behind an office or degree (`Juez`, `Magistrado`, `Doctora en Derecho.`) and 1 not a name (`f`, a one-word cut refused since). The population for Spanish titles in front of a name. The DOF's texts are not protected by copyright (LFDA art. 14 VIII), and SIDOF permits reproduction without profit with the source cited, which the credit line says | the office before the name, which the cut drops, so a snapshot reads names with nothing before them, and the comma; names a person typed themselves, since the DOF's typesetters write every one; private persons, a signatory being an official, a judge or a party's representative; names outside Mexico and in any script but Latin. The `title` field reads an abbreviation in capitals by shape, so a lone initial counts as one (`J. Guadalupe`, José's), as would a short surname in capitals ending a sentence; one in mixed case only from `local_dof._ABBREVIATED`'s list, so `Ma.` (María's) does not | A live source has no fixed snapshot to pin, so the cached copy is the pin: `meta.json` holds the source's data date and the sha256 of `raw.jsonl` and `corpus.jsonl`, checked on every load, and every report, the `name_usage.py` row included, carries the source and that date. Quote a count with the date, and expect a later fetch to differ. A local snapshot records which fetch it was read from, and `snapshot.py --local` refuses to compare it with a corpus fetched since, which would report the names the source gained and lost as moves: after a `fetch --refresh`, record again from the base checkout. ClinicalTrials.gov's terms ask anyone redistributing its data to credit it, show NLM's processing date and say what was changed; keeping the names in the cache and citing counts with that header meets them. The FEC's bar on soliciting and commercial use is in the credit line of every `fec-individuals` report, so a count quoted from it carries the bar with it. A record there is a distinct name, not a contribution or a contributor: one donor gives many times (34,264,742 rows held 1,893,964 names in the file dated 2026-10-18), and counting contributions would weight a shape by how often its bearers give. `tools/name_usage.py` looks a word up in each corpus's word index, cached as `wordindex.json` beside it on first use and keyed to the records' sha256, to the code that cuts and folds a word and to the record keys it reads (`Source.corpus_keys`; a record whose two names write one word counts once), so a refetch, a change to the fold or a change to which of a record's names are read rebuilds it; a single-word lookup it answers parses neither the records nor the names. Measured 2026-10-22 with both corpora fetched, a single-word `name_usage.py --no-wikidata` run took 2.5s with their indexes cached (12.5s before the cache, 14.8s on the run that builds it). A spaced word (`Ph. D.`) still scans every name, which took 8.6s, nearly all of it the FEC corpus's. For `orcid-researchers` and `patentsview-inventors`, whose extracts keep every record's boxes and country uncapped, `name_usage.py --boxes` adds a row by box and country, read from a second index, `boxindex.json.gz` beside the corpus (`cached_box_index`): per word and country the people whose given box or family box writes it, each country shown as a share of its people, so a word can be weighed in the country a language pack is for (`--country CC`). It is keyed to the extract's sha256, taken from the bytes as they are read, and to a fingerprint of the reader and the fold, so a refetch rebuilds it. Building it took 2 min 19 s for ORCID's 24,990,909 records (3,780,309 words in 6,660,756 word-and-country cells, 41 MB cached) on 2026-10-28. With both indexes cached a lookup took 10.4 s with `--boxes` against 5.3 s without, the difference nearly all reading the two indexes (the 369 MB extract hashes in a fraction of a second once the system has it cached), which is why the rows are asked for rather than always built; an extract that cannot be read fails its own row, naming the refresh. `fetch --refresh` re-downloads the FEC's zip rather than re-reading the cached one, replacing it only once the new one is whole, and a corpus fetched for another cycle than `FEC_CYCLE` is refused on load. The `fec-individuals` fetch refuses a file whose lines stop matching the 21 columns it reads, or whose names stop holding a comma, rather than read it as a smaller or a different corpus. A `ukec-individuals` record is a distinct name too, not a donation or a `DonorId`: one name often holds several IDs, which seem to come with reports rather than with people (15,214 names held 24,473 on 2026-10-22, the IDs counted in `meta.json`'s detail; 2,663 names held several at one `RegulatedEntityId` alone, counted from the export and not kept, though some of those may be two people of one name), so a count of IDs would weight a name by how often it was reported. Its fetch reads the register's total from the JSON search under the same filter before and after the export, and refuses an export holding another number of rows, or a register that changed in between: the export ignores `rows=`, so the total is what tells a cut or capped export from the whole register. It refuses an export whose last row is cut short (Python's `csv` returns a body cut inside a row as a short last row, which the row count would otherwise take for a whole one), and a header other than the 29 columns `UKEC_COLUMNS` names, and counts into `meta.json`'s detail the rows it skips: rows not 29 fields, rows from a donor that is not an individual (the filter asks the server to leave those out), and individuals with no name. Past `UKEC_MAX_BAD` of either of the first two, or `UKEC_MAX_NAMELESS` of the third (4 of 42,355 rows were nameless on 2026-10-22), it refuses the export. The data date is when the first search answered, in UTC: the register states no date of its own, so the counts above, dated by the day they were fetched (2026-10-22), carry the data date 2026-10-23T03:13:19Z in their header. `tw-donors` keeps one zip per report under `reports/`, whole, its masked ID, address and phone columns included, as the FEC's zip is kept: the platform's download answers a report that does not exist and a fault alike, with a 500 and an incident number, so the fetch reads which reports exist from the platform's listing and refuses a listed report it cannot download. It reads each file's columns by name, since a 2020 report lacks a column the newest ones write, and refuses a column it does not know, a category it does not know, a row whose serial column names another report (the zip is not the one asked for), a listing that changes during the fetch, and a report past `TW_MAX_REPORT_BAD` of whose rows it reads past. It reads back the rows a stray quote took into one field (169 rows in 12 records of three reports on 2026-10-22, 8 of them two rows joined by two stray quotes), counts donor cells that hold no name (匿名, a dash) or a masked one, and caches a downloaded zip only once its rows are read, so a refused zip is asked for again. A zip the platform rebuilt under the name it had is not seen: delete `reports/` to download every report again. A layout check written from one surveyed file meets every other file only at fetch time: `tw-donors`' first three fetches were each refused partway, by a report with no income writing one row of dashes, by a 2020 report with 24 columns where the surveyed one had 25, and by a row with no category. Each refusal cost every download again until the source cached its zips, so a new source that downloads many files caches each one before its checks run. `tse-donors` caches nothing but the corpus: each election's national member (`receitas_candidatos__BRASIL.csv`) is read from the TSE's CDN in HTTP Range requests, 209 MB for the two elections where their zips are 1.82 GB (2026-10-22), because the TSE regenerates the zips every few days and a cached one would be stale within the week. Every range request carries `If-Range` with the ETag the HEAD gave if it is strong, else its `Last-Modified`, and every part must state the `Content-Range` asked for, so a zip replaced during the read is refused rather than read across two versions; a HEAD that states no length, neither a strong ETag nor a `Last-Modified`, or `Accept-Ranges` other than `bytes` is refused before the read. A fetch took 69s on 2026-10-22. It refuses a zip without the national member, a header that is not 60 columns or lacks its three columns, a `DS_ORIGEM_RECEITA` value it does not know (a renamed category, or the file no longer in Latin-1), malformed rows or lines folded into a field past `TSE_MAX_BAD`, an election with too few rows from individuals (`TSE_MIN_INDIVIDUALS`: a member cut to its header, or the donations moved under another category) too many without a name or registry name (`TSE_MAX_MISSING`, counting a placeholder such as `#NULO` or `***` as none: a masked column) or with a number in one (`TSE_MAX_NUMBERED`: a numeric column moved under a name), too few distinct filed or registry names (`TSE_MIN_DISTINCT`: a column masked with one word), and a corpus fetched for other elections than `TSE_ELECTIONS`. The CPF in each row is unmasked in the file, and like every other column it is read past, never stored or printed; a pair either of whose names holds more than `TSE_MAX_DIGITS` digits is dropped and counted, since campaigns sometimes typed a number, a CPF among them, into the name field. A record is a pair, so the counts in its reports (`tally`, the `name_usage.py` row) are of pairs, and a filed name standing beside two registry names counts twice. diff --git a/tools/differential/local_dof.py b/tools/differential/local_dof.py index e38b1ea6..7e8276b2 100644 --- a/tools/differential/local_dof.py +++ b/tools/differential/local_dof.py @@ -75,13 +75,20 @@ #: meta.json's detail: a change must bump it, and a corpus cut the old #: way is refused on load (check_dof). No refetch is needed: the cut is #: read from the cached days. -DOF_CUT_VERSION = 4 +DOF_CUT_VERSION = 5 #: The blocks kept before each block holding a "Rúbrica": the stacked #: signature's office lines and name, or a date line before them. EXCERPT_BLOCKS = 3 -#: Gates: listed notes with HTML whose note came back without it (or -#: flagged too large to send), and "Rúbrica"s no rule could cut. +#: Gates, each a share past which the fetch is refused: notes listed with +#: HTML that state a size and come back without it (22 of 23,816 in +#: 2025); notes listed with HTML that state a size of 0, published as page +#: images (2,936 of 26,752, 11.0%), gated in case an ordinary note comes +#: to state 0; weekdays listing no note at all (7 of 261, each a public +#: holiday); and "Rúbrica"s no rule could cut (1,741 of 35,305 under cut +#: version 5). Measured 2026-10-31. DOF_MAX_EMPTY = 0.05 +DOF_MAX_IMAGE_ONLY = 0.20 +DOF_MAX_SILENT = 0.05 DOF_MAX_UNCUT = 0.10 _DAYS = "days" @@ -183,17 +190,30 @@ def excerpts(content: str) -> list[list[tuple[str, str]]]: # -- cutting a name ---------------------------------------------------------- #: The spelled-out titles a name may open with, as the DOF writes them, -#: matched case-insensitively. An abbreviated title is told by its shape -#: instead (_ABBREVIATION). +#: matched case-insensitively. An abbreviated title in capitals is told by +#: its shape (_CAPITALS), one in mixed case by this list (_ABBREVIATED). SPELLED_TITLES = frozenset({ "licenciado", "licenciada", "maestro", "maestra", "doctor", "doctora", "ingeniero", "ingeniera", "arquitecto", "arquitecta", "contador", "contadora", "ciudadano", "ciudadana", "profesor", "profesora", "actuario", "actuaria", "notario", "notaria"}) -#: An abbreviated title: one or more chunks of a capital and up to four -#: lower-case letters, each ending in a full stop (`Lic.`, `C.`, `C.P.`, -#: `L.C.P.`, `Mtra.`, `Lcda.`), or capitals alone (`LIC.`). -_ABBREVIATION = re.compile(r"(?:[A-ZÁÉÍÓÚÑ](?:[a-záéíóúñ]{0,4}|[A-ZÁÉÍÓÚÑ]{0,4})\.)+") +#: An abbreviated title in capitals: one or more chunks of up to five +#: capitals, each ending in a full stop (`LIC.`, `C.`, `C.P.`, `L.C.P.`, +#: `MVZ.`), the degrees too many to list. It is a shape, so a lone +#: initial passes (`J. Guadalupe`, José's), as does a short surname in +#: capitals ending a sentence. +_CAPITALS = re.compile(r"(?:[A-ZÁÉÍÓÚÑ]{1,5}\.)+") +#: The abbreviated titles in mixed case, without the full stop they are +#: written with: the courtesy and academic titles and the ranks the DOF's +#: 2025 signatures open with. A list, not a shape, since a capital and a +#: few letters with a full stop is also a given name written short (`Ma.`, +#: `Gpe.`) and a short surname ending a sentence (`Ruiz Paz. Conste`). +_ABBREVIATED = frozenset({ + "lic", "licda", "licdo", "lcda", "lcdo", "mtro", "mtra", "mtr", "dr", "dra", + "ing", "arq", "prof", "profr", "profa", "act", "abog", "biól", "biol", "geóg", + "psic", "med", "esp", "adm", "urb", "sen", "dip", "mag", "gral", "div", "brig", + "bgda", "big", "cor", "tte", "sbtte", "cap", "alm", "almte", "nav", "cab", "pil", + "cmdte", "magap", "ret"}) # `ret`: retirado, inside a rank (`Alm. Ret.`) #: An article before a title or a name (`la Lic. ...`, `el C. ...`), #: left out of the cut. _ARTICLES = frozenset({"el", "la", "los", "las"}) @@ -201,11 +221,24 @@ def excerpts(content: str) -> list[list[tuple[str, str]]]: # noviembre de dos mil veinticinco`, `30 DE MAYO DE 2025`), and an office # signing without a name (`EL # SECRETARIO DE SALUD`), which opens with an article where a name does -# not. A bare digit or month is not enough: `CAP. 1/O. AYTE.` is a rank +# not (_office() has the rest). A bare digit or month is not enough: `CAP. 1/O. AYTE.` is a rank # and `Julio` a given name; `Del Olmo` is a surname, so `del` is not an # article here. _MONTHS = ("enero|febrero|marzo|abril|mayo|junio|julio|agosto|septiembre|setiembre" "|octubre|noviembre|diciembre") +#: The head words of the offices that sign without a name, read only in +#: a cut wholly in capitals (`DIRECTORA GENERAL DE RECURSOS HUMANOS`, `C. +#: GOBERNADOR CONSTITUCIONAL DEL ESTADO`): in mixed case they open a name +#: as an office written as a title (`Magistrado José ...`, `Jueza de +#: Distrito Ana ...`), which is kept. +_OFFICE_HEADS = frozenset({ + "director", "directora", "subdirector", "subdirectora", "gobernador", "gobernadora", + "secretario", "secretaria", "subsecretario", "subsecretaria", "presidente", + "presidenta", "titular", "jefe", "jefa", "coordinador", "coordinadora", "delegado", + "delegada", "encargado", "encargada", "procurador", "procuradora", "fiscal", + "comisionado", "comisionada", "consejero", "consejera", "diputado", "diputada", + "senador", "senadora", "tesorero", "tesorera", "oficial", "administrador", + "administradora", "gerente", "representante", "apoderado", "apoderada"}) _DATE = re.compile(rf"\bde (?:{_MONTHS})\b|\bdos mil\b|\b\d{{4}}\b", re.I) #: The end of the sentence before a signature: `.-` or `.–`. _CLAUSE_START = re.compile(r"\.\s*[-–—]\s*") @@ -213,15 +246,31 @@ def excerpts(content: str) -> list[list[tuple[str, str]]]: #: full stop, with the spaces around it. _BEFORE_RUBRICA = re.compile(r"(?:\.\s*)?[-–—]?\s*$") _EDGE = " ,;:.-–—\"'()" -#: The most words a title rule's cut may hold: the longest names in 400 -#: cuts checked by hand ran to 7 (`CAP. 1/O. AYTE. CONTAB. DANIEL LARA +#: The most words a title rule's cut may hold: the longest name among the +#: 800 cuts checked by hand on 2026-10-31 (200 drawn at random under each +#: of cut versions 1, 2, 3 and 5) ran to 7 (`CAP. 1/O. AYTE. CONTAB. DANIEL LARA #: HERNANDEZ`). _TITLE_WORDS = 8 +def _office(name: str) -> bool: + """Whether a cut is an office signing without a name: one opening + with an article (`EL SECRETARIO DE SALUD`), or, in capitals, with an + office's head word after any titles or with `DE`/`DEL` (`DEL ESTADO`; + in mixed case `Del Olmo` is a surname).""" + words = name.split() + if words[0].lower() in _ARTICLES: + return True + if not name.isupper(): + return False + rest = [w for w in words if not is_title(w)] or words + return rest[0].lower() in _OFFICE_HEADS or words[0] in ("DE", "DEL") + + def is_title(word: str) -> bool: """Whether `word` is a title a name may open with, by cut()'s test.""" - return (bool(_ABBREVIATION.fullmatch(word)) + return (bool(_CAPITALS.fullmatch(word)) + or (word.endswith(".") and word[:-1].lower() in _ABBREVIATED) or word.lower().strip(".") in SPELLED_TITLES) @@ -252,10 +301,15 @@ def cut(text: str, mask: str) -> tuple[str, str] | None: """The name in a signature's clause and the rule that cut it, or None when no rule finds one. The rules, in order: - - "bold": the last run of words with a bold letter, the run ending - at a word that ends in a comma (`Pardo, Presidenta ...` keeps - `Pardo`), with any titles directly before it (`Lic.`, which the - DOF mostly leaves in roman); + - "bold": the last run of words with a bold letter, starting after + the last bold word that ends in a comma, semicolon or colon, so + that of a run in bold throughout the part after it is kept (`GRAL. + BRIG. E.M., Ana Ruiz Paz` keeps `Ana Ruiz Paz`; in 2025 the part + after the comma was the name in about 78 of the 90 signatures + holding one, measured 2026-10-31 on cut version 4, the rest an + office after the name, which _office() mostly refuses), with any + titles directly before it (`Lic.`, which the DOF mostly leaves in + roman); - "title": no bold, so the run from the last title (is_title) with a word behind it, to the first comma after it (`Sen. Ana Ruiz, Secretaria` keeps `Sen. Ana Ruiz`), and no longer than @@ -358,7 +412,7 @@ def signatures(excerpt: list[tuple[str, str]]) -> Iterator[dict[str, str] | str] if _DATE.search(name): yield f"{layout}, cut a date" continue - if name.split()[0].lower() in _ARTICLES: + if _office(name): yield f"{layout}, cut an office" continue clause = prefix + name_text @@ -384,13 +438,16 @@ def _day_path(day: datetime.date) -> Path: def _get(url: str) -> dict[str, Any]: - """`url`'s JSON object, required to answer messageCode 200.""" - data = _retried(url, lambda body: corpus_build.parse_json(body, url), DOF_DELAY, - timeout=120) - if not isinstance(data, dict) or data.get("messageCode") != 200: - code = data.get("messageCode") if isinstance(data, dict) else type(data).__name__ - raise ValueError(f"{url} answered {code}, not messageCode 200") - return data + """`url`'s JSON object, required to answer messageCode 200; another + answer is retried like a body cut short, and refused once the retries + run out.""" + def read(body: bytes) -> dict[str, Any]: + data = corpus_build.parse_json(body, url) + if not isinstance(data, dict) or data.get("messageCode") != 200: + code = data.get("messageCode") if isinstance(data, dict) else type(data).__name__ + raise ValueError(f"answered {code}, not messageCode 200") + return data + return _retried(url, read, DOF_DELAY, timeout=120) def disallowed(robots: str) -> frozenset[int]: @@ -412,11 +469,19 @@ def disallowed(robots: str) -> frozenset[int]: def _fetch_day(day: datetime.date, skip: frozenset[int]) -> dict[str, Any]: """A day's listing and every listed note with HTML, read into excerpts; the counts read past are kept with them.""" - listing = _get(DOF_DAY_URL.format(day=day.strftime("%d-%m-%Y"))) + url = DOF_DAY_URL.format(day=day.strftime("%d-%m-%Y")) + listing = _get(url) + lacking = [e for e in DOF_EDITIONS if not isinstance(listing.get(e), list)] + if lacking: + # Every 2025 listing holds all three, a list each, an empty one + # on a day without that edition; a day read from a listing + # without one would be cached short for good. + raise ValueError(f"{url} lists no {', '.join(lacking)}: the listing's " + f"layout may have changed under the reader") notes: list[dict[str, Any]] = [] counts: Counter[str] = Counter() for edition in DOF_EDITIONS: - for entry in listing.get(edition) or []: + for entry in listing[edition]: cod = int(entry["codNota"]) if entry.get("existeHtml") != "S": counts["without html"] += 1 @@ -460,13 +525,16 @@ def _read_day(path: Path) -> dict[str, Any] | None: def _cache_days(*, again: bool = False) -> list[dict[str, Any]]: """Every day of DOF_FROM..DOF_TO, from the cache where it holds the day (unless `again`), else fetched and cached.""" - skip = disallowed(_retried(DOF_ROBOTS_URL, lambda b: b.decode("utf-8"), DOF_DELAY)) + skip: frozenset[int] | None = None # robots.txt, read once a day needs fetching (directory(DOF) / _DAYS).mkdir(exist_ok=True) out = [] for number, day in enumerate(_days(), 1): path = _day_path(day) held = None if again else _read_day(path) if held is None: + if skip is None: + skip = disallowed(_retried(DOF_ROBOTS_URL, lambda b: b.decode("utf-8"), + DOF_DELAY)) held = _fetch_day(day, skip) body = (corpus_build.jsonl_line(held, sort_keys=True) + "\n").encode() corpus_build.write_atomic(path, gzip.compress(body, mtime=0)) @@ -481,11 +549,11 @@ def fetch_dof() -> Fetched: cached): the cut name, its rule, title and join (signatures()), the clause, the day and the note. - A listed note that states a size but comes back without HTML (one - stating 0 is image only, and read past), or a "Rúbrica" no rule - cut, is counted, and past DOF_MAX_EMPTY or DOF_MAX_UNCUT of - the notes or the "Rúbrica"s the fetch is refused: a changed layout - would otherwise read as a year with fewer signatures.""" + A listed note that states a size but comes back without HTML, one + stating 0 (image only), a weekday listing no note, and a "Rúbrica" + no rule cut are counted, and past DOF_MAX_EMPTY, DOF_MAX_IMAGE_ONLY, + DOF_MAX_SILENT or DOF_MAX_UNCUT the fetch is refused: a changed + layout would otherwise read as a year with fewer signatures.""" days = _cache_days() records: list[dict[str, str]] = [] counts: Counter[str] = Counter() @@ -500,8 +568,15 @@ def fetch_dof() -> Fetched: continue records.append({**found, "day": day["day"], "note": str(note["note"])}) listed = counts["read"] + counts["empty"] + with_html = listed + counts["image only"] + weekdays = [day for day in days + if datetime.date.fromisoformat(day["day"]).weekday() < 5] + silent = sum(1 for day in weekdays if not sum(day["counts"].values())) marks = len(records) + sum(uncut.values()) _refuse_past([(counts["empty"], listed, DOF_MAX_EMPTY, "notes with HTML came back empty"), + (counts["image only"], with_html, DOF_MAX_IMAGE_ONLY, + "notes with HTML stated a size of 0"), + (silent, len(weekdays), DOF_MAX_SILENT, "weekdays listed no note"), (sum(uncut.values()), marks, DOF_MAX_UNCUT, "Rúbricas cut no name")], "the DOF's layout may have changed under the reader") return Fetched(records, DOF_TO.isoformat(), { @@ -527,6 +602,12 @@ def renew_dof() -> None: "version {wanted}; fetch again, which reads the cached days") +_check_excerpt = _pinned("excerpt_version", lambda: DOF_EXCERPT_VERSION, + "was cut from excerpts of version {held}, and this tool reads " + "version {wanted}; fetch again, which reads every day again") + + def check_dof(meta: dict[str, Any]) -> str: - """Why a corpus of other days or another cut is refused.""" - return _check_range(meta) or _check_to(meta) or _check_cut(meta) + """Why a corpus of other days, other excerpts or another cut is refused.""" + return (_check_range(meta) or _check_to(meta) or _check_excerpt(meta) + or _check_cut(meta)) From 52d60b4dc598a721d0fbcd1e51938c615f066619 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Sat, 31 Oct 2026 04:39:09 -0700 Subject: [PATCH 3/3] Keep a capitalised name behind an office word in the dof cut MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit _office() refused any cut in capitals opening with an office's head word, which dropped CONSEJERO FRANCISCO RAMOS SILVA. It now refuses one only where the head word is the whole cut or is followed by DE, DEL, GENERAL, CONSTITUCIONAL or another head word: the one name comes back and the 52 offices stay refused (cut version 6: 33,565 signatures). cut()'s docstring now says what _office() does with the bold-comma non-names: it refuses one, and keeps an office in mixed case (3) and the motto's NO REELECCIÓN (2). The README gives both stacked shares: 45.5% are stacked, 45.3% under an office in the block above. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- tests/v2/test_local_dof.py | 3 +++ tools/differential/README.md | 2 +- tools/differential/local_dof.py | 30 +++++++++++++++++++++--------- 4 files changed, 26 insertions(+), 11 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 0344346e..7e748dbe 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -256,7 +256,7 @@ The 2.0 rewrite landed as underscore-private modules beside the v1 code, which 2 **Adding a curated sub-set of an existing vocabulary set** (a marker over a base set, as `AFTER_GIVEN_TITLES` is over `TITLES`) — it is a behavioral PROPERTY of a reading, not a second word list: a row in `nameparser/_reading_index.py`'s `BEHAVIORAL` table, a branch in its `compile_readings` adding the words whose reading carries it and a key in its `Compiled` TypedDict, a constant of the same name in `tools/vocab/build.py`'s `CONSTANTS` roster (an import-time assert there holds the two to one list), and the `NAME = _compiled.NAME` assignment in the config module. A property the compiler reads is kept in the reading index by `indexed()`, which reads `BEHAVIORAL`, so a new one reaches a choice of languages too. The subset relation then holds by construction, since the property cannot exist without its reading, so it needs no check. A marker that records a clash with a NAME is the other shape: derived from the word's name reading beside the clashing one (`SUFFIX_ACRONYMS_AMBIGUOUS`, `SUFFIX_WORDS_AMBIGUOUS`, and the complement `NON_GIVEN_NAME_PARTICLES` works from), not written as a property, so it too holds by construction (decisions/vocabulary-model, 2026-10-25). A relation construction does not give (`∩ == ∅` with a set the sub-set cannot logically overlap; a sub-set member that is also a title is silently inert, title handling consuming it first) goes in `check_pool`, with a mutation in `tests/v2/test_vocab.py`'s table. Do **not** forbid `titles ∩ particles` — that overlap is intentional (`freiherr`, `st`). A caller's `Lexicon.add()` to a sub-set does **not** add the word to its base, so a caller adding a brand-new word adds it to both. Two traps a gating set met (#752 phase 2i). Where a caller listing a word in the BASE set has always meant "read it as this everywhere" (the 1.4 recipe `c.titles.add("dean")`), the gated words compile OUT of the base rather than into both, or the gate overrides the caller's listing: `TITLES_AMBIGUOUS` is not a subset of `TITLES` for that reason, and a word in both reads as a title. And `_VOCAB_FIELDS` is read whole by two rosters in `_pipeline/_vocab.py`, the capitals shape's "unlisted" (`in_any_wordlist`) and the comma contrast's "claimed as non-name" (`_NON_NAME_FIELDS`), so a new field joins both by default; a field whose words may be names leaves both, which the grid found as `John Smith, DEAN` reversed where C6 had read a credential; #789's `given_name_abbreviations` met it again (`John Smith, JAS` and `Jas SMITH, XYZ` reversed), found by review because its grid held only the dotted spellings the set was for -- a new field's grid holds the spellings it must NOT move too. -**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials --base `, which reads the base from a baseline kept for its package's files and never the shared snapshot beside the corpus (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating; the shared snapshot beside the corpus is one file per source for every worktree, and #752's phase 2f found ctgov reporting a sibling branch's 1,334 joined-credential readings as its own moves against it); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to English titles, honorifics or post-nominals also reads `snapshot.py --local govuk-people` the same way: GOV.UK's display names of ministers, officials and officers, the title run and the post-nominals as an office writes them (`The Rt Hon`, `Sir`, `Baroness`, ranks; `KC`, `CBE`, `KCB`), in 2,093 of whose 6,021 `parse()` read one or both on 2026-10-30. A change to Spanish titles, to the `es` pack, or to how a title marks where a name starts after an office reads `snapshot.py --local dof-signatures` the same way: the names in the signature blocks of Mexico's Diario Oficial for 2025, cut from the clause that names the signatory's office, 65.4% of its 33,564 signatures opening with a title (`Lic.`, `Dr.`, `Mtro.`, `C.`, `Licenciada`, `C.P.`), whose first word `parse()` read as a title in 82.1% and `parser_for(locales.ES)` in 86.8% on 2026-10-31. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific titles such as `Guru` in front, `Shekh`, a title there until #752's phase 2j made it one only inside a title phrase, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals --base ` the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality; and `local_agreement.py orcid-researchers --base `, ORCID holders' own two boxes (given names, family name) from every country, scored by the country they work or live in -- a population each person split for themselves, academic but worldwide, the one to read for a change a language pack makes; and `local_agreement.py patentsview-inventors --base `, US patents' inventors in the two boxes their filings give, split by a professional filer, with how many people filed under each name in each country, the one to read for how common a name's word is among a country's US-patent inventors. A change to titles or post-nominals, or to the `cs` pack, reads `local_agreement.py cz-candidates --base `, and with `--locale cs` too: Czech election candidates' four boxes (the title before the name, given names, surname, the title after it), scored by election, the only population whose titles are stated apart from the name, 36% of its 170,181 records titled (2026-10-30), nearly all academic degrees; its names are composed from the boxes (`Ing. Jan Novák, Ph.D.`), so it says nothing about how a writer spaces or punctuates them. These are the only real fielded populations outside the committed fields files, and no CI run sees them either. +**Validating a new parsing rule** — before implementing, simulate it in a throwaway script against `TEST_NAMES` (plus a few target-language examples) to catch regressions/false-positives early. E.g. this surfaced the `last_base` `do`/`st`/`mc` empties and the patronymic `"David Michael Abramovich"` false-reorder. After the change, `tests/v2/test_snapshot.py` lists every corpus name it moved, grouped by changed field -- read them, and re-record with `uv run python tools/differential/snapshot.py --record` once every move is intended, so the PR diff of `snapshot.jsonl` shows the reviewer the same list. `uv run python tools/differential/cldr_agreement.py` reports how the tree's agreement with CLDR's stated name fields moved; `tests/v2/test_cldr_agreement.py` fails on any move, and `tools/differential/cldr_agreement.json` doubles as a measured list of names the parser and CLDR read differently. The same command and test score the GND corpus's real names (`gnd_fields.jsonl`, record `gnd_agreement.json`) by the same rules, and `--record` rewrites every record, printing per source what it accepts. A locale-pack change reads the second view the same command prints, per pack (`cldr_agreement_locales.json`, #672, and `wikidata_native_agreement_locales.json` for real Han and kana names); its `JA` rows run only with the `ja` extra installed. A change to Han, Hangul or kana order or to Korean segmentation reads `wikidata_native_agreement.json` too: the only real names in those scripts the corpora hold. It holds no honorific, so a change to the honorific peel moves nothing there. A change to comma or credential handling also reads the local corpus of hand-typed names (`snapshot.py --local ctgov-officials --base `, which reads the base from a baseline kept for its package's files and never the shared snapshot beside the corpus (a base read once from the same fetch is copied rather than parsed again; `--sample N` reads a fixed sample while iterating; the shared snapshot beside the corpus is one file per source for every worktree, and #752's phase 2f found ctgov reporting a sibling branch's 1,334 joined-credential readings as its own moves against it); tools/differential/README.md's "Local corpora"): the one population of names nobody chose for a test or a catalog, so it shows how often each shape is typed, and the one no CI run sees. A change to title handling reads `ukec-individuals` the same way: UK donors' names in spoken order and mixed case, in 8,219 of whose 15,214 `parse()` read a title on 2026-10-22. A change to English titles, honorifics or post-nominals also reads `snapshot.py --local govuk-people` the same way: GOV.UK's display names of ministers, officials and officers, the title run and the post-nominals as an office writes them (`The Rt Hon`, `Sir`, `Baroness`, ranks; `KC`, `CBE`, `KCB`), in 2,093 of whose 6,021 `parse()` read one or both on 2026-10-30. A change to Spanish titles, to the `es` pack, or to how a title marks where a name starts after an office reads `snapshot.py --local dof-signatures` the same way: the names in the signature blocks of Mexico's Diario Oficial for 2025, cut from the clause that names the signatory's office, 65.4% of its 33,565 signatures opening with a title (`Lic.`, `Dr.`, `Mtro.`, `C.`, `Licenciada`, `C.P.`), whose first word `parse()` read as a title in 82.1% and `parser_for(locales.ES)` in 86.8% on 2026-10-31. A change to particles or generational words reads `snapshot.py --local tse-donors` the same way, Brazilian names in spoken order dense with both. A change to initials, to a title word outside English, or to how a script's combining marks are read reads `snapshot.py --local nepal-candidates` the same way: Nepal's local election candidates, each a romanized name (surnames written as initials, `B.K.`, `K.C.`; honorific titles such as `Guru` in front, `Shekh`, a title there until #752's phase 2j made it one only inside a title phrase, and `Md.`, which its period makes a title by shape) beside the same name in Devanagari, written with vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and with the joiners ZWJ and ZWNJ. It is the only large population of South Asian names the corpora hold, and its corpus reads both scripts. A change to non-Latin scripts or to Arabic-script names reads `snapshot.py --local jrc-names` the same way, and `local_corpora.py groups jrc-names`: the spellings of one person across scripts and transliterations, harvested from news text, with 17,074 Arabic-script spellings where the committed corpora hold 315 distinct names with an Arabic letter, 96 of them CLDR's fictional renderings (2026-10-24), and under a licence that keeps even a reading of one of its names in the cache. A change to which field a word lands in also reads `local_agreement.py psc-individuals --base ` the same way: real names with the four boxes their filers typed them into (UK companies' people with significant control), scored field by field and by nationality; and `local_agreement.py orcid-researchers --base `, ORCID holders' own two boxes (given names, family name) from every country, scored by the country they work or live in -- a population each person split for themselves, academic but worldwide, the one to read for a change a language pack makes; and `local_agreement.py patentsview-inventors --base `, US patents' inventors in the two boxes their filings give, split by a professional filer, with how many people filed under each name in each country, the one to read for how common a name's word is among a country's US-patent inventors. A change to titles or post-nominals, or to the `cs` pack, reads `local_agreement.py cz-candidates --base `, and with `--locale cs` too: Czech election candidates' four boxes (the title before the name, given names, surname, the title after it), scored by election, the only population whose titles are stated apart from the name, 36% of its 170,181 records titled (2026-10-30), nearly all academic degrees; its names are composed from the boxes (`Ing. Jan Novák, Ph.D.`), so it says nothing about how a writer spaces or punctuates them. These are the only real fielded populations outside the committed fields files, and no CI run sees them either. ## Gotchas diff --git a/tests/v2/test_local_dof.py b/tests/v2/test_local_dof.py index eab3dea7..e455d6ab 100644 --- a/tests/v2/test_local_dof.py +++ b/tests/v2/test_local_dof.py @@ -172,6 +172,9 @@ def test_a_run_in_bold_throughout_keeps_the_part_after_its_last_comma() -> None: ("Jueza de Distrito Ana Lis Mora", False), ("Del Olmo Blanco", False), # a surname ("LIC. ANA LIS MORA", False), + ("CONSEJERO FRANCISCO RAMOS SILVA", False), # an office before a name, in capitals + ("DIPUTADA SECRETARIA", True), + ("GOBERNADORA DEL ESTADO", True), ]) def test_an_office_signing_without_a_name(name: str, office: bool) -> None: assert dof._office(name) is office diff --git a/tools/differential/README.md b/tools/differential/README.md index 1a85e2a6..d67fcd40 100644 --- a/tools/differential/README.md +++ b/tools/differential/README.md @@ -304,7 +304,7 @@ Any snapshot reading of 5,000 entries or more, committed or local, parses in a w | `jrc-names` | the person entities of the EU Joint Research Centre's JRC-Names, each with the spellings of its name harvested from news text in many languages and scripts: a record is one distinct spelling, with its entity and its script. The file (`entities.gzip`, 14.8 MB) is pinned by sha256 and dated by its Last-Modified, 2024-03-15. Of its 1,317,586 spellings kept, 1,203,634 are Latin, 66,673 Cyrillic, 17,074 Arabic (7,833 persons, 5,347 of them spelled in Latin too), 13,039 Katakana and 8,016 Han, and 218,822 persons have two or more spellings, 25,415 of them in two scripts or more (a spelling whose letters are in two scripts, or none, not counted as one). The population for spelling and transliteration variants of one name, read by the `groups` view, and for Arabic-script names. Licensed under the JRC-Names EULA, personal and non-transferable, with no redistribution or derivative works, so its views' output and its snapshot stay in the cache like every local corpus's, and here the licence, not only the names being real people's, is why | titles and credentials, since the names are harvested from running text (`parse()` read a title in 3,011 of its 1,315,051 names and a suffix in 1,940, 2026-10-24); the ASCII comma, which none holds (1,510 Arabic spellings hold an Arabic comma, U+060C, which the parser reads as one); the Indic scripts (Devanagari 827 spellings, Bengali 419, the others fewer); people the news did not name | | `nepal-candidates` | every candidate in Nepal's local elections of 2079 BS (2022), from the Election Commission's results site (result.election.gov.np), one file per local body: a record per distinct pair of the Commission's romanization (`name`, the one `parse()` reads) and the name in Devanagari as filed (`devanagari`). The corpus and the word index hold both, so a snapshot and a `name_usage.py` lookup read both scripts; `tally`, which counts what follows a comma, reads the romanized name. Of 126,981 pairs fetched 2026-10-24 (152,960 candidates in 753 files, 26 of whose romanizations were a placeholder such as `- - -` and are dropped), the romanized names carry surnames written as initials (`B.K.`, `K.C.`, `G.C.`), honorific words that are titles in front (`Guru`; `Shekh` until #752's phase 2j made it a title only in a title phrase) and `Md.`, Muhammad's abbreviation, which its period makes a title by shape; the Devanagari ones carry vowel signs and the virama, combining marks the letter-shape tests do not count as letters (decisions/nfd-shapes), and the joiners ZWJ and ZWNJ. The population for romanized South Asian names and for Devanagari. The site states no licence, which the credit line says | case and the comma, since the Commission's romanization is one consistent hand; names a person typed themselves; credentials and generational words; the parliament and provincial candidates, whose files carry Devanagari only; names outside Nepal | | `cz-candidates` | every candidate on the registers the Czech Statistical Office (ČSÚ) publishes on volby.gov.cz for the 2022 municipal elections (`kv2022`) and the 2025 Chamber of Deputies election (`ps2025`): a record is a distinct candidate's four name boxes in one election, the title before the name (`TITULPRED`), the given names (`JMENO`), the surname (`PRIJMENI`) and the title after it (`TITULZA`), with the election; 165,736 records of kv2022's 197,506 candidates and 4,445 of ps2025's 4,475 on 2026-10-30. A record's name is COMPOSED, not written by anyone: the boxes joined in the Czech convention, `TITULPRED JMENO PRIJMENI, TITULZA` (`Ing. Jan Novák, Ph.D.`). 61,320 of its 170,181 records carry a title (58,667 before the name, 7,687 after it), nearly all academic degrees (`Ing.`, `Mgr.`, `Bc.`, `MUDr.`; `DiS.`, `Ph.D.`, `MBA`): the population for titles and post-nominals the filer kept apart from the name. Scored by `local_agreement.py` by election, under the default parser and under `--locale cs`. ČSÚ licenses it under CC BY 4.0, asking that derived data be marked as derived, which the composed names are and the credit line says | case, spacing and punctuation of a writer's own, since the string is composed; the family-comma shape; courtesy titles and offices; names outside Czechia; the elections not in `VOLBY_ELECTIONS` (the index lists kv2002 to kv2026 and ps2002 to ps2025) | -| `dof-signatures` | the signatures in the notes of Mexico's Diario Oficial de la Federación (DOF) for 2025, read through SIDOF's web services (a listing per day, each note's HTML; the notes its robots.txt disallows are skipped), each name cut from the clause that signs it (`local_dof.cut`): the last run of bold words, with the titles before it, which the DOF mostly sets in roman, or with no bold the run a title opens, up to a comma. A record is one signature (a signatory signs many notes): the name, the rule that cut it, the layout (`stacked`, the office on lines above the name and `Rúbrica.` on a line below; `inline`, `Office, Title Name.- Rúbrica.-` in a paragraph), the title the name opens with, how it meets the text before it, and the clause. Of the 35,305 `Rúbrica`s in 2025's notes (fetched 2026-10-31: of the 26,752 notes listed with HTML, 2,936 were published as page images, stating a size of 0, and 22 came back blank though they stated one), 33,564 were cut, to 6,643 distinct names, and the 1,741 left are counted by reason in `meta.json` (neither bold nor a title, an office signing without a name, a date, one word); every day's listing was read again on 2026-10-31 and held all three editions and as many notes as cached, the 7 weekdays listing none being public holidays. Of the signatures, 65.4% open with a title (`Lic.` 27.7%, `Dr.`/`Dra.` 9.3%, `Mtro.`/`Mtra.` 8.2%, `C.` 5.1%, `Ing.` 3.6%, a spelled-out `Licenciado`/`Licenciada` 2.9%, a dotted acronym degree such as `C.P.` or `L.C.P.` in most of the rest), and 45.3% stand under their office in a stacked block, glued to it with no comma once the lines are read as one. Of 200 cuts drawn at random and checked by hand on 2026-10-31 under cut version 5, the one fetched, 199 were exactly a name and 1 a name behind a rank (`VICEALMIRANTE`), none not a name; under cut version 3, 196, 3 behind an office or degree (`Juez`, `Magistrado`, `Doctora en Derecho.`) and 1 not a name (`f`, a one-word cut refused since). The population for Spanish titles in front of a name. The DOF's texts are not protected by copyright (LFDA art. 14 VIII), and SIDOF permits reproduction without profit with the source cited, which the credit line says | the office before the name, which the cut drops, so a snapshot reads names with nothing before them, and the comma; names a person typed themselves, since the DOF's typesetters write every one; private persons, a signatory being an official, a judge or a party's representative; names outside Mexico and in any script but Latin. The `title` field reads an abbreviation in capitals by shape, so a lone initial counts as one (`J. Guadalupe`, José's), as would a short surname in capitals ending a sentence; one in mixed case only from `local_dof._ABBREVIATED`'s list, so `Ma.` (María's) does not | +| `dof-signatures` | the signatures in the notes of Mexico's Diario Oficial de la Federación (DOF) for 2025, read through SIDOF's web services (a listing per day, each note's HTML; the notes its robots.txt disallows are skipped), each name cut from the clause that signs it (`local_dof.cut`): the last run of bold words, with the titles before it, which the DOF mostly sets in roman, or with no bold the run a title opens, up to a comma. A record is one signature (a signatory signs many notes): the name, the rule that cut it, the layout (`stacked`, the office on lines above the name and `Rúbrica.` on a line below; `inline`, `Office, Title Name.- Rúbrica.-` in a paragraph), the title the name opens with, how it meets the text before it, and the clause. Of the 35,305 `Rúbrica`s in 2025's notes (fetched 2026-10-31: of the 26,752 notes listed with HTML, 2,936 were published as page images, stating a size of 0, and 22 came back blank though they stated one), 33,565 were cut, to 6,644 distinct names, and the 1,740 left are counted by reason in `meta.json` (neither bold nor a title, an office signing without a name, a date, one word); every day's listing was read again on 2026-10-31 and held all three editions and as many notes as cached, the 7 weekdays listing none being public holidays. Of the signatures, 65.4% open with a title (`Lic.` 27.7%, `Dr.`/`Dra.` 9.3%, `Mtro.`/`Mtra.` 8.2%, `C.` 5.1%, `Ing.` 3.6%, a spelled-out `Licenciado`/`Licenciada` 2.9%, a dotted acronym degree such as `C.P.` or `L.C.P.` in most of the rest), and 45.5% are stacked, 45.3% under their office in the block above, glued to it with no comma once the lines are read as one (the rest with no block above the name in its excerpt). Of 200 cuts drawn at random and checked by hand on 2026-10-31 under cut version 5, 199 were exactly a name and 1 a name behind a rank (`VICEALMIRANTE`), none not a name; under cut version 3, 196, 3 behind an office or degree (`Juez`, `Magistrado`, `Doctora en Derecho.`) and 1 not a name (`f`, a one-word cut refused since). The population for Spanish titles in front of a name. The DOF's texts are not protected by copyright (LFDA art. 14 VIII), and SIDOF permits reproduction without profit with the source cited, which the credit line says | the office before the name, which the cut drops, so a snapshot reads names with nothing before them, and the comma; names a person typed themselves, since the DOF's typesetters write every one; private persons, a signatory being an official, a judge or a party's representative; names outside Mexico and in any script but Latin. The `title` field reads an abbreviation in capitals by shape, so a lone initial counts as one (`J. Guadalupe`, José's), as would a short surname in capitals ending a sentence; one in mixed case only from `local_dof._ABBREVIATED`'s list, so `Ma.` (María's) does not | A live source has no fixed snapshot to pin, so the cached copy is the pin: `meta.json` holds the source's data date and the sha256 of `raw.jsonl` and `corpus.jsonl`, checked on every load, and every report, the `name_usage.py` row included, carries the source and that date. Quote a count with the date, and expect a later fetch to differ. A local snapshot records which fetch it was read from, and `snapshot.py --local` refuses to compare it with a corpus fetched since, which would report the names the source gained and lost as moves: after a `fetch --refresh`, record again from the base checkout. ClinicalTrials.gov's terms ask anyone redistributing its data to credit it, show NLM's processing date and say what was changed; keeping the names in the cache and citing counts with that header meets them. The FEC's bar on soliciting and commercial use is in the credit line of every `fec-individuals` report, so a count quoted from it carries the bar with it. A record there is a distinct name, not a contribution or a contributor: one donor gives many times (34,264,742 rows held 1,893,964 names in the file dated 2026-10-18), and counting contributions would weight a shape by how often its bearers give. `tools/name_usage.py` looks a word up in each corpus's word index, cached as `wordindex.json` beside it on first use and keyed to the records' sha256, to the code that cuts and folds a word and to the record keys it reads (`Source.corpus_keys`; a record whose two names write one word counts once), so a refetch, a change to the fold or a change to which of a record's names are read rebuilds it; a single-word lookup it answers parses neither the records nor the names. Measured 2026-10-22 with both corpora fetched, a single-word `name_usage.py --no-wikidata` run took 2.5s with their indexes cached (12.5s before the cache, 14.8s on the run that builds it). A spaced word (`Ph. D.`) still scans every name, which took 8.6s, nearly all of it the FEC corpus's. For `orcid-researchers` and `patentsview-inventors`, whose extracts keep every record's boxes and country uncapped, `name_usage.py --boxes` adds a row by box and country, read from a second index, `boxindex.json.gz` beside the corpus (`cached_box_index`): per word and country the people whose given box or family box writes it, each country shown as a share of its people, so a word can be weighed in the country a language pack is for (`--country CC`). It is keyed to the extract's sha256, taken from the bytes as they are read, and to a fingerprint of the reader and the fold, so a refetch rebuilds it. Building it took 2 min 19 s for ORCID's 24,990,909 records (3,780,309 words in 6,660,756 word-and-country cells, 41 MB cached) on 2026-10-28. With both indexes cached a lookup took 10.4 s with `--boxes` against 5.3 s without, the difference nearly all reading the two indexes (the 369 MB extract hashes in a fraction of a second once the system has it cached), which is why the rows are asked for rather than always built; an extract that cannot be read fails its own row, naming the refresh. `fetch --refresh` re-downloads the FEC's zip rather than re-reading the cached one, replacing it only once the new one is whole, and a corpus fetched for another cycle than `FEC_CYCLE` is refused on load. The `fec-individuals` fetch refuses a file whose lines stop matching the 21 columns it reads, or whose names stop holding a comma, rather than read it as a smaller or a different corpus. A `ukec-individuals` record is a distinct name too, not a donation or a `DonorId`: one name often holds several IDs, which seem to come with reports rather than with people (15,214 names held 24,473 on 2026-10-22, the IDs counted in `meta.json`'s detail; 2,663 names held several at one `RegulatedEntityId` alone, counted from the export and not kept, though some of those may be two people of one name), so a count of IDs would weight a name by how often it was reported. Its fetch reads the register's total from the JSON search under the same filter before and after the export, and refuses an export holding another number of rows, or a register that changed in between: the export ignores `rows=`, so the total is what tells a cut or capped export from the whole register. It refuses an export whose last row is cut short (Python's `csv` returns a body cut inside a row as a short last row, which the row count would otherwise take for a whole one), and a header other than the 29 columns `UKEC_COLUMNS` names, and counts into `meta.json`'s detail the rows it skips: rows not 29 fields, rows from a donor that is not an individual (the filter asks the server to leave those out), and individuals with no name. Past `UKEC_MAX_BAD` of either of the first two, or `UKEC_MAX_NAMELESS` of the third (4 of 42,355 rows were nameless on 2026-10-22), it refuses the export. The data date is when the first search answered, in UTC: the register states no date of its own, so the counts above, dated by the day they were fetched (2026-10-22), carry the data date 2026-10-23T03:13:19Z in their header. `tw-donors` keeps one zip per report under `reports/`, whole, its masked ID, address and phone columns included, as the FEC's zip is kept: the platform's download answers a report that does not exist and a fault alike, with a 500 and an incident number, so the fetch reads which reports exist from the platform's listing and refuses a listed report it cannot download. It reads each file's columns by name, since a 2020 report lacks a column the newest ones write, and refuses a column it does not know, a category it does not know, a row whose serial column names another report (the zip is not the one asked for), a listing that changes during the fetch, and a report past `TW_MAX_REPORT_BAD` of whose rows it reads past. It reads back the rows a stray quote took into one field (169 rows in 12 records of three reports on 2026-10-22, 8 of them two rows joined by two stray quotes), counts donor cells that hold no name (匿名, a dash) or a masked one, and caches a downloaded zip only once its rows are read, so a refused zip is asked for again. A zip the platform rebuilt under the name it had is not seen: delete `reports/` to download every report again. A layout check written from one surveyed file meets every other file only at fetch time: `tw-donors`' first three fetches were each refused partway, by a report with no income writing one row of dashes, by a 2020 report with 24 columns where the surveyed one had 25, and by a row with no category. Each refusal cost every download again until the source cached its zips, so a new source that downloads many files caches each one before its checks run. `tse-donors` caches nothing but the corpus: each election's national member (`receitas_candidatos__BRASIL.csv`) is read from the TSE's CDN in HTTP Range requests, 209 MB for the two elections where their zips are 1.82 GB (2026-10-22), because the TSE regenerates the zips every few days and a cached one would be stale within the week. Every range request carries `If-Range` with the ETag the HEAD gave if it is strong, else its `Last-Modified`, and every part must state the `Content-Range` asked for, so a zip replaced during the read is refused rather than read across two versions; a HEAD that states no length, neither a strong ETag nor a `Last-Modified`, or `Accept-Ranges` other than `bytes` is refused before the read. A fetch took 69s on 2026-10-22. It refuses a zip without the national member, a header that is not 60 columns or lacks its three columns, a `DS_ORIGEM_RECEITA` value it does not know (a renamed category, or the file no longer in Latin-1), malformed rows or lines folded into a field past `TSE_MAX_BAD`, an election with too few rows from individuals (`TSE_MIN_INDIVIDUALS`: a member cut to its header, or the donations moved under another category) too many without a name or registry name (`TSE_MAX_MISSING`, counting a placeholder such as `#NULO` or `***` as none: a masked column) or with a number in one (`TSE_MAX_NUMBERED`: a numeric column moved under a name), too few distinct filed or registry names (`TSE_MIN_DISTINCT`: a column masked with one word), and a corpus fetched for other elections than `TSE_ELECTIONS`. The CPF in each row is unmasked in the file, and like every other column it is read past, never stored or printed; a pair either of whose names holds more than `TSE_MAX_DIGITS` digits is dropped and counted, since campaigns sometimes typed a number, a CPF among them, into the name field. A record is a pair, so the counts in its reports (`tally`, the `name_usage.py` row) are of pairs, and a filed name standing beside two registry names counts twice. diff --git a/tools/differential/local_dof.py b/tools/differential/local_dof.py index 7e8276b2..a7c199f2 100644 --- a/tools/differential/local_dof.py +++ b/tools/differential/local_dof.py @@ -75,7 +75,7 @@ #: meta.json's detail: a change must bump it, and a corpus cut the old #: way is refused on load (check_dof). No refetch is needed: the cut is #: read from the cached days. -DOF_CUT_VERSION = 5 +DOF_CUT_VERSION = 6 #: The blocks kept before each block holding a "Rúbrica": the stacked #: signature's office lines and name, or a date line before them. EXCERPT_BLOCKS = 3 @@ -84,8 +84,8 @@ #: 2025); notes listed with HTML that state a size of 0, published as page #: images (2,936 of 26,752, 11.0%), gated in case an ordinary note comes #: to state 0; weekdays listing no note at all (7 of 261, each a public -#: holiday); and "Rúbrica"s no rule could cut (1,741 of 35,305 under cut -#: version 5). Measured 2026-10-31. +#: holiday); and "Rúbrica"s no rule could cut (1,740 of 35,305 under cut +#: version 6). Measured 2026-10-31. DOF_MAX_EMPTY = 0.05 DOF_MAX_IMAGE_ONLY = 0.20 DOF_MAX_SILENT = 0.05 @@ -239,6 +239,10 @@ def excerpts(content: str) -> list[list[tuple[str, str]]]: "comisionado", "comisionada", "consejero", "consejera", "diputado", "diputada", "senador", "senadora", "tesorero", "tesorera", "oficial", "administrador", "administradora", "gerente", "representante", "apoderado", "apoderada"}) +#: What follows an office's head word in an office (`DIRECTORA GENERAL`, +#: `GOBERNADOR CONSTITUCIONAL`, `SECRETARIA DE ...`), besides another head +#: word (`DIPUTADA SECRETARIA`). +_OFFICE_NEXT = frozenset({"DE", "DEL", "GENERAL", "CONSTITUCIONAL"}) _DATE = re.compile(rf"\bde (?:{_MONTHS})\b|\bdos mil\b|\b\d{{4}}\b", re.I) #: The end of the sentence before a signature: `.-` or `.–`. _CLAUSE_START = re.compile(r"\.\s*[-–—]\s*") @@ -255,16 +259,22 @@ def excerpts(content: str) -> list[list[tuple[str, str]]]: def _office(name: str) -> bool: """Whether a cut is an office signing without a name: one opening - with an article (`EL SECRETARIO DE SALUD`), or, in capitals, with an - office's head word after any titles or with `DE`/`DEL` (`DEL ESTADO`; - in mixed case `Del Olmo` is a surname).""" + with an article (`EL SECRETARIO DE SALUD`), or, in capitals, with + `DE`/`DEL` (`DEL ESTADO`; in mixed case `Del Olmo` is a surname) or + with an office's head word after any titles that is the whole cut or + is followed by _OFFICE_NEXT or another head word.""" words = name.split() if words[0].lower() in _ARTICLES: return True if not name.isupper(): return False + if words[0] in ("DE", "DEL"): + return True rest = [w for w in words if not is_title(w)] or words - return rest[0].lower() in _OFFICE_HEADS or words[0] in ("DE", "DEL") + # The head word alone, or one an office's next word follows; a name + # may follow it too (`CONSEJERO FRANCISCO RAMOS SILVA`), and is kept. + return rest[0].lower() in _OFFICE_HEADS and ( + len(rest) == 1 or rest[1] in _OFFICE_NEXT or rest[1].lower() in _OFFICE_HEADS) def is_title(word: str) -> bool: @@ -306,8 +316,10 @@ def cut(text: str, mask: str) -> tuple[str, str] | None: that of a run in bold throughout the part after it is kept (`GRAL. BRIG. E.M., Ana Ruiz Paz` keeps `Ana Ruiz Paz`; in 2025 the part after the comma was the name in about 78 of the 90 signatures - holding one, measured 2026-10-31 on cut version 4, the rest an - office after the name, which _office() mostly refuses), with any + holding one, measured 2026-10-31 on cut version 4; of the rest, + _office() refuses one, and an office in mixed case after the name + (`Director de Substanciación ...`, 3) and the motto's `NO + REELECCIÓN` (2) are kept), with any titles directly before it (`Lic.`, which the DOF mostly leaves in roman); - "title": no bold, so the run from the last title (is_title) with