RegexPair.matchmake crashed with a TypeError when a capture-group
pattern was applied to a revision whose text or comment was deleted or
suppressed (content is None). Guard both matching paths against None,
and make the no-capture-group path emit a None column for such
revisions instead of omitting the key entirely.
This carries forward the fix Kaylea Champion and Mako Hill made on the
mako_changes-20230429 branch (7e6cd5b), which predated the rewrite.
Adds a unit test for matchmake(None) and an end-to-end test against the
ikwiki dump, which contains revisions with deleted text and comments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit dbcae5c64e)
Per-page adaptive parse skip: once mwparserfromhell times out on a
revision of size N bytes, force-timeout future revisions on the same
page whose text is >= N bytes instead of waiting another 60s on each
one. State resets per page, so well-behaved pages are unaffected.
Motivated by zhwiki bot-vandalism log pages (e.g. ns=4
'Wikipedia:当前的破坏/...' at 4.6GB across 6133 revisions) where every
revision triggers the 60s SIGALRM, blocking the worker for hours.
- Implement per-namespace resume points (dict mapping namespace -> (pageid, revid))
to correctly handle interleaved dump ordering in partitioned output
- Extract resume functionality to dedicated resume.py module
- Add graceful shutdown handling via shutdown_requested flag (CLI-level only)
- Use lazy ParquetWriter creation to avoid empty files on early exit
- Refactor writing logic to _write_batch() helper method
- Simplify control flow by replacing continue statements with should_write flag
