Skip to content

Add the dof-signatures local corpus: DOF signature blocks cut to names - #977

Merged
derek73 merged 3 commits into
masterfrom
feat/dof-signatures-corpus
Oct 31, 2026
Merged

derek73 merged 3 commits into
masterfrom
feat/dof-signatures-corpus

Conversation

@derek73

@derek73 derek73 commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner

What

A new unscored local corpus, dof-signatures (tools/differential/local_dof.py). It holds the names in the signature blocks of Mexico's Diario Oficial de la Federación for 2025, read through SIDOF's JSON web services. It is cached under ~/.cache/nameparser-tools/local/ like the other local corpora and never committed, because the signatories are real officials.

It is registered in local_corpora.py's SOURCES. A row is added to the README's Local corpora table, and AGENTS.md's "Validating a new parsing rule" gains a sentence saying when to read it: a change to Spanish titles, to the es pack, or to how a title marks where a name starts after an office.

Terms

  • LFDA art. 14 VIII (Ley Federal del Derecho de Autor): administrative texts are not protected by copyright. Quoted: "No son objeto de la protección como derecho de autor … VIII. Los textos legislativos, reglamentarios, administrativos o judiciales".
  • SIDOF's Aviso Legal: "La reproducción total o parcial … sin fines de lucro es permitida siempre y cuando se cite la fuente." The credit line cites it.
  • The API is the one SIDOF offers on its open-data page. robots.txt's disallowed notes are skipped.

The cut

Each "Rúbrica" in a note's HTML is read in one of two layouts:

  • stacked: office lines, then the name, then Rúbrica. on a line of its own;
  • inline: Office, Title Name.- Rúbrica.- in a paragraph.

The name is the last run of bold words, plus the titles before it, which the DOF mostly sets in roman. With no bold, it is the run a title opens, up to a comma and at most 8 words.

A run in bold throughout keeps the part after its last comma (GRAL. BRIG. E.M., Ana Ruiz Paz), which was the name in about 78 of the 90 such signatures. A mixed-case abbreviation counts as a title only if it is on a list, so Ma. (María) and Paz. Conste don't. Capitals keep the shape test.

A "Rúbrica" nothing cuts is counted by reason in meta.json. Each of these refusals was measured on the year before it was added:

  • neither bold nor a title;
  • an office signing without a name: one opening with an article (EL SECRETARIO DE SALUD, 389 signatures), or, in capitals, with DE/DEL or an office's head word that stands alone or before another office word (DIRECTORA GENERAL …, 52; CONSEJERO FRANCISCO RAMOS SILVA is kept);
  • a date;
  • a one-word cut.

The first draft's "what follows the last comma" and "whole clause" fallbacks were retired. They cut a name in 2 of 46 and about 5 of 60 distinct outputs.

The fetch caches one gzip of excerpts per day, so it resumes after an interruption and the cut can be re-read offline; robots.txt is read only once a day needs fetching. DOF_CUT_VERSION triggers a re-read and DOF_EXCERPT_VERSION a refetch, and the load check pins both.

The fetch refuses on any of these:

  • a listing without all three editions;
  • more than 5% of notes that state a size coming back blank (22 in 2025);
  • more than 20% of notes stating a size of 0, the page-image notes (2,936 of 26,752, 11.0%);
  • more than 5% of weekdays listing no note (7 of 261, all holidays);
  • more than 10% of "Rúbrica"s left uncut.

A messageCode other than 200 is retried. Every 2025 listing was read again: all held the three editions and as many notes as cached.

Measured (2026-10-31)

  • 33,565 signatures, 6,644 distinct names, from 35,305 "Rúbrica"s (cut version 6).
  • Precision: of 200 random cuts checked by hand under cut version 5 (version 6 restores one capitalised name it refused), 199 were exactly a name and 1 a name behind a rank (VICEALMIRANTE …); none was not a name.
  • Titled share: 65.4% of signatures open with a title.
    • Lic. 27.7%, Dr./Dra. 9.3%, Mtro./Mtra. 8.2%, C. 5.1%, Ing. 3.6%.
    • Spelled-out Licenciado/Licenciada 2.9%, and multi-period acronym degrees (C.P., L.C.P., M.A.P. …) 2.9%.
  • Office glued with no comma: 45.5% of signatures are stacked, 45.3% under their office in the block above, and another 1.3% follow it inline with no punctuation.
  • Readings: parse() reads the first word of a titled signature as a title in 82.1%; parser_for(locales.ES) does so in 86.8%. The misses are a leading C. (1,708 signatures), multi-period acronym degrees (988; their one-period spellings, LCP., already read as titles) and Maestro/Maestra (85). Issue drafts are with the session, not filed.

Checks

  • uv run pytest, uv run mypy and uv run ruff check all clean.
  • tests/v2/test_local_dof.py: 70 offline tests; each guard added in review fails against the module before it, with invented names and a fake urlopen.
  • The new tests also pass on 3.15.
  • One real fetch of the full year, and snapshot.py --local dof-signatures recorded and matching.

🤖 Generated with Claude Code

A new unscored local corpus, tools/differential/local_dof.py: the
signatures in the 2025 notes of Mexico's Diario Oficial de la
Federación, read through SIDOF's web services (a listing per day, each
note's HTML; notes robots.txt disallows are skipped). Each name is cut
from the clause that signs it -- the last run of bold words with the
titles before it, or with no bold the run a title opens -- and the
record keeps the clause, the rule, the layout, the title and how the
name meets the text before it. Cached and never committed: the
signatories are real officials.

The fetch caches one gzip of excerpts per day, so it resumes and the
cut is re-read offline: DOF_CUT_VERSION re-reads, DOF_EXCERPT_VERSION
refetches. It refuses past 5% of stated-size notes coming back blank
(image-only notes, which state size 0, are counted apart) and past 10%
of "Rúbrica"s left uncut, each uncut one counted by reason.

Measured 2026-10-31: 33,623 signatures, 6,663 distinct names, 65.3%
opening with a title; 196 of 200 hand-checked cuts exactly a name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
derek73 and others added 2 commits October 31, 2026 04:32
- check_dof pins excerpt_version beside cut_version, so a corpus cut
  from excerpts of another version is refused on load.
- A day listing lacking any of the three editions, or holding one that
  is not a list, is refused rather than cached short; and a gate refuses
  past 5% of weekdays listing no note (7 of 261 in 2025, each a public
  holiday; every 2025 listing read again held all three editions and as
  many notes as cached).
- Image-only notes (stated size 0, 11.0% in 2025) are gated at 20%.
- cut()'s docstring now says what the bold rule does: of a run in bold
  throughout, the part after its last comma is kept (the name in about
  78 of 90 such signatures). A cut wholly in capitals opening with an
  office's head word or with DE/DEL is refused as an office (52
  signatures; one name lost, CONSEJERO ...).
- robots.txt is read only once a day needs fetching, so a re-cut asks
  nothing of the site, as the docstring said.
- A mixed-case abbreviation is a title only from a list, so `Ma.`,
  `Gpe.`, `Eds.` and a surname ending a sentence (`Paz. Conste`) no
  longer are; capitals keep the shape.
- A messageCode other than 200 is retried like a body cut short.
- The hand-check counts are dated and keyed to their cut versions.

Cut version 5: 33,564 signatures, 6,643 distinct names; of 200 cuts
checked by hand, 199 exactly a name and 1 a name behind a rank.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_office() refused any cut in capitals opening with an office's head
word, which dropped CONSEJERO FRANCISCO RAMOS SILVA. It now refuses one
only where the head word is the whole cut or is followed by DE, DEL,
GENERAL, CONSTITUCIONAL or another head word: the one name comes back
and the 52 offices stay refused (cut version 6: 33,565 signatures).

cut()'s docstring now says what _office() does with the bold-comma
non-names: it refuses one, and keeps an office in mixed case (3) and
the motto's NO REELECCIÓN (2). The README gives both stacked shares:
45.5% are stacked, 45.3% under an office in the block above.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant