Repository navigation
Add the dof-signatures local corpus: DOF signature blocks cut to names - #977
Merged
Merged
Conversation
A new unscored local corpus, tools/differential/local_dof.py: the signatures in the 2025 notes of Mexico's Diario Oficial de la Federación, read through SIDOF's web services (a listing per day, each note's HTML; notes robots.txt disallows are skipped). Each name is cut from the clause that signs it -- the last run of bold words with the titles before it, or with no bold the run a title opens -- and the record keeps the clause, the rule, the layout, the title and how the name meets the text before it. Cached and never committed: the signatories are real officials. The fetch caches one gzip of excerpts per day, so it resumes and the cut is re-read offline: DOF_CUT_VERSION re-reads, DOF_EXCERPT_VERSION refetches. It refuses past 5% of stated-size notes coming back blank (image-only notes, which state size 0, are counted apart) and past 10% of "Rúbrica"s left uncut, each uncut one counted by reason. Measured 2026-10-31: 33,623 signatures, 6,663 distinct names, 65.3% opening with a title; 196 of 200 hand-checked cuts exactly a name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- check_dof pins excerpt_version beside cut_version, so a corpus cut from excerpts of another version is refused on load. - A day listing lacking any of the three editions, or holding one that is not a list, is refused rather than cached short; and a gate refuses past 5% of weekdays listing no note (7 of 261 in 2025, each a public holiday; every 2025 listing read again held all three editions and as many notes as cached). - Image-only notes (stated size 0, 11.0% in 2025) are gated at 20%. - cut()'s docstring now says what the bold rule does: of a run in bold throughout, the part after its last comma is kept (the name in about 78 of 90 such signatures). A cut wholly in capitals opening with an office's head word or with DE/DEL is refused as an office (52 signatures; one name lost, CONSEJERO ...). - robots.txt is read only once a day needs fetching, so a re-cut asks nothing of the site, as the docstring said. - A mixed-case abbreviation is a title only from a list, so `Ma.`, `Gpe.`, `Eds.` and a surname ending a sentence (`Paz. Conste`) no longer are; capitals keep the shape. - A messageCode other than 200 is retried like a body cut short. - The hand-check counts are dated and keyed to their cut versions. Cut version 5: 33,564 signatures, 6,643 distinct names; of 200 cuts checked by hand, 199 exactly a name and 1 a name behind a rank. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_office() refused any cut in capitals opening with an office's head word, which dropped CONSEJERO FRANCISCO RAMOS SILVA. It now refuses one only where the head word is the whole cut or is followed by DE, DEL, GENERAL, CONSTITUCIONAL or another head word: the one name comes back and the 52 offices stay refused (cut version 6: 33,565 signatures). cut()'s docstring now says what _office() does with the bold-comma non-names: it refuses one, and keeps an office in mixed case (3) and the motto's NO REELECCIÓN (2). The README gives both stacked shares: 45.5% are stacked, 45.3% under an office in the block above. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A new unscored local corpus,
dof-signatures(tools/differential/local_dof.py). It holds the names in the signature blocks of Mexico's Diario Oficial de la Federación for 2025, read through SIDOF's JSON web services. It is cached under~/.cache/nameparser-tools/local/like the other local corpora and never committed, because the signatories are real officials.It is registered in
local_corpora.py'sSOURCES. A row is added to the README's Local corpora table, and AGENTS.md's "Validating a new parsing rule" gains a sentence saying when to read it: a change to Spanish titles, to theespack, or to how a title marks where a name starts after an office.Terms
robots.txt's disallowed notes are skipped.The cut
Each "Rúbrica" in a note's HTML is read in one of two layouts:
Rúbrica.on a line of its own;Office, Title Name.- Rúbrica.-in a paragraph.The name is the last run of bold words, plus the titles before it, which the DOF mostly sets in roman. With no bold, it is the run a title opens, up to a comma and at most 8 words.
A run in bold throughout keeps the part after its last comma (
GRAL. BRIG. E.M., Ana Ruiz Paz), which was the name in about 78 of the 90 such signatures. A mixed-case abbreviation counts as a title only if it is on a list, soMa.(María) andPaz. Constedon't. Capitals keep the shape test.A "Rúbrica" nothing cuts is counted by reason in
meta.json. Each of these refusals was measured on the year before it was added:EL SECRETARIO DE SALUD, 389 signatures), or, in capitals, withDE/DELor an office's head word that stands alone or before another office word (DIRECTORA GENERAL …, 52;CONSEJERO FRANCISCO RAMOS SILVAis kept);The first draft's "what follows the last comma" and "whole clause" fallbacks were retired. They cut a name in 2 of 46 and about 5 of 60 distinct outputs.
The fetch caches one gzip of excerpts per day, so it resumes after an interruption and the cut can be re-read offline; robots.txt is read only once a day needs fetching.
DOF_CUT_VERSIONtriggers a re-read andDOF_EXCERPT_VERSIONa refetch, and the load check pins both.The fetch refuses on any of these:
A
messageCodeother than 200 is retried. Every 2025 listing was read again: all held the three editions and as many notes as cached.Measured (2026-10-31)
VICEALMIRANTE …); none was not a name.Lic.27.7%,Dr./Dra.9.3%,Mtro./Mtra.8.2%,C.5.1%,Ing.3.6%.Licenciado/Licenciada2.9%, and multi-period acronym degrees (C.P.,L.C.P.,M.A.P.…) 2.9%.parse()reads the first word of a titled signature as a title in 82.1%;parser_for(locales.ES)does so in 86.8%. The misses are a leadingC.(1,708 signatures), multi-period acronym degrees (988; their one-period spellings,LCP., already read as titles) andMaestro/Maestra(85). Issue drafts are with the session, not filed.Checks
uv run pytest,uv run mypyanduv run ruff checkall clean.tests/v2/test_local_dof.py: 70 offline tests; each guard added in review fails against the module before it, with invented names and a fakeurlopen.snapshot.py --local dof-signaturesrecorded and matching.🤖 Generated with Claude Code