Brings in revision-level redirect detection and its regenerated
baselines. Conflicts were the expected adjacent insertions where both
branches extended the same seams (build_table and WikiqParser
signatures, the argparse block, the build_table call sites, and the
test file); resolved by keeping both branches' additions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The redirect_target column added in 59fea19 was page-level: the page's
state at dump time, stamped onto every revision row of the page. That
invited the wrong inference that a given revision was a redirect, and
the column was empty for dumps whose export never wrote <redirect>
elements, such as the 2023 wikitravel.org scrapes; a Swedish Wikitravel
scrape with 732 in-text redirect revisions produced no redirect signal
at all.
Remove that column and detect redirects from each revision's own text
instead. revision_is_redirect records whether the text begins with a
redirect directive and revision_redirect_target records the directive's
link target with any fragment and label stripped. #REDIRECT is
recognized on every wiki; localized keywords (e.g. OMDIRIGERING on
Swedish wikis) can be added with --redirect-aliases. Revisions with
deleted or unavailable text get nulls in both columns. The redirect-map
use case behind 59fea19 survives: the last revision's
revision_redirect_target per page reconstructs the page-level map, now
also on dumps without <redirect> elements.
Document that title and namespace are page-level identity values as of
the time of export, not historical facts about each revision.
Regenerate the test baselines for the column change. Every regenerated
file was verified to differ from its predecessor only by removing
redirect_target and adding the two new columns, with identical values
in all shared columns.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A single mwparserfromhell tree walk per revision emits two nullable
int64 columns. prose_chars counts rendered content outside any template
or tag: text nodes, wikilink labels (or targets when unlabeled),
external link labels, and heading titles. structured_chars counts the
same kinds of content inside template parameter values or tag contents,
with each character assigned by its nearest enclosing container. Markup
syntax, template and tag names, parameter names, attributes, bare URLs,
and HTML comments count in neither, so syntax overhead is derivable as
the total size minus the two columns.
Deleted revisions and parse timeouts yield nulls through the same
machinery as the other parser-based columns and do not change which
revisions are processed. Note that mwparserfromhell parses bold and
italic markup as tags, so emphasized text counts as structured.
Includes hand-computed fixture tests for the counting rules and an
end-to-end test on a dump with deleted revisions verifying nulls and
the invariant prose_chars + structured_chars <= revision size.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
No test reads this file; the regex tests use the basic_regextest and
capturegroup_regextest baselines.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
assert_equal_enough wrote its inputs to files named "token" and "rev"
in the current directory on every invocation, littering the repository
root whenever the test suite ran.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Port the regex counting feature from the mako_changes-20230429 branch
(2ff4d60): the -RPc/--revision-pattern-count and
-CPc/--comment-pattern-count flags cause revision and comment patterns
to output the number of matches instead of the matched text, with 0
(never null) for revisions with no matches or deleted content. The
flags apply to all revision or comment patterns respectively, and count
columns are typed int64 in the output schema.
Counting the matches of common or large patterns previously required
returning the full matched text, which made for very large outputs.
Also take the code review suggestion from that branch (933ca75): use
assignment expressions so matching no longer makes redundant calls to
search() before finditer()/findall() or calls match.group() twice.
Includes unit tests for the count semantics and schema types, and an
end-to-end test that verifies counts against the number of matches
recomputed from each revision's text and comment.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resume with collapsed revision groups was untested: the resume point is
the (articleid, revid) of the last written row, which for collapsed
output is the last revision of a group, and nothing verified that the
replay reconstructs group boundaries and collapsed_revs counts
identically across the resume point. Run sailormoon with
--collapse-user, truncate the output at the midpoint, resume, and
assert the result is identical to an uninterrupted run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test_infobox reads test/test_diff_revisions/test_infobox_from and
test_infobox_to, which were never committed to the repository, so the
test always failed with FileNotFoundError. Skip it when the fixtures
are missing so it runs again if they are ever added.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Commit 59fea19 added a redirect_target column to wikiq output but did
not regenerate the test baselines, leaving 14 baseline-comparison tests
failing. Regenerate the affected baselines from current output. Each
regenerated file was verified to differ from its old baseline only by
the addition of the new column: row counts and all values in shared
columns are identical.
Also add the noargs_sailormoon.jsonl baseline used by test_jsonl_noargs,
which was never committed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
build_table returns (table, reverts_column, wikitext_parser) but the
jsonl test helper still unpacked two values, so test_jsonl_noargs and
test_jsonl_tsv_equivalence failed with a ValueError before reading any
output. Also update the docstring, which still described the two-value
return.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Nate has given up on getting wikiq to output parquet directly because it
makes too many unpredictable large memory allocations. The workflow now
outputs JSONL in a single pass and then uses spark (wikiq_spark) to index
it as parquet in a second pass.
Remove the parquet output path along with the machinery that existed only
to support it: checkpoint files, resume temp-file merging, namespace
partitioning, and file rotation (--partition-namespaces,
--max-revisions-per-file). Resume support remains for JSONL output, where
the resume point is derived from the last complete line of the output
file. Also remove the parquet tests and baseline files.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RegexPair.matchmake crashed with a TypeError when a capture-group
pattern was applied to a revision whose text or comment was deleted or
suppressed (content is None). Guard both matching paths against None,
and make the no-capture-group path emit a None column for such
revisions instead of omitting the key entirely.
This carries forward the fix Kaylea Champion and Mako Hill made on the
mako_changes-20230429 branch (7e6cd5b), which predated the rewrite.
Adds a unit test for matchmake(None) and an end-to-end test against the
ikwiki dump, which contains revisions with deleted text and comments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Implement per-namespace resume points (dict mapping namespace -> (pageid, revid))
to correctly handle interleaved dump ordering in partitioned output
- Extract resume functionality to dedicated resume.py module
- Add graceful shutdown handling via shutdown_requested flag (CLI-level only)
- Use lazy ParquetWriter creation to avoid empty files on early exit
- Refactor writing logic to _write_batch() helper method
- Simplify control flow by replacing continue statements with should_write flag
The big challenges here (and remaining) are as follows:
1. Deltas requires changes to be given at the token level,
whereas wikidiff2 reports changes at the byte level. Thus,
it is often required to tokenize sequences of text to convert
to the desired token indices. As-is this is done inefficiently,
often requiring re-tokenization of previously-tokenized sequences.
A better implementation would incrementally tokenize, or
automatically find the referenced sequences.
2. Deltas only allows for Equal/Insert/Delete operations,
while wikidiff2 also detects paragraph moves. These paragraph
moves are NOT equivalent to Equal, as the moved paragraphs
are not guaranteed to be equivalent, just very similar.
Wikidiff2 does not report changes to moved paragraphs, so
to preserve token persistence, a difference algorithm
would need to be performed on the before/after sequences.
A stopgap (currently implemented) is to turn these
into strict deletions/insertions.
3. There appears to be a lot of memory consumption, and
sometimes this results in memory overflow. I am unsure
if this is a memory leak or simply that re-tokenizing
causes significant enough memory throughput that
my machine can't handle it.
4. Deltas expects all tokens in the before/after text to
be covered by segment ranges of Equal/Insert/Delete, but
wikidiff2 does not appear to ever emit any Equal ranges,
instead skipping them. These ranges must be computed
and inserted in sequence. As-is the code does not correctly
handle unchanged text at the end of pages.
Signed-off-by: Will Beason <willbeason@gmail.com>
This is inefficient as it requires an individal request per diff.
Going to try collecting the revision texts to reduce communication
overhead.
Signed-off-by: Will Beason <willbeason@gmail.com>
