With pywikidiff2 and mediawiki-utilities no longer installed by default,
the tests covering --diff, -p wikidiff2, and -p legacy cannot run on a
base install. Mark them so a base install reports skips rather than
failures, and keep the markers in wikiq_test_utils.py so all three test
modules share one definition of what each feature needs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 59d5a9d46a04320bad5d81e0edf5ecd11f9c2a9b)
Nate has given up on getting wikiq to output parquet directly because it
makes too many unpredictable large memory allocations. The workflow now
outputs JSONL in a single pass and then uses spark (wikiq_spark) to index
it as parquet in a second pass.
Remove the parquet output path along with the machinery that existed only
to support it: checkpoint files, resume temp-file merging, namespace
partitioning, and file rotation (--partition-namespaces,
--max-revisions-per-file). Resume support remains for JSONL output, where
the resume point is derived from the last complete line of the output
file. Also remove the parquet tests and baseline files.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 72a8851373)
Resume with collapsed revision groups was untested: the resume point is
the (articleid, revid) of the last written row, which for collapsed
output is the last revision of a group, and nothing verified that the
replay reconstructs group boundaries and collapsed_revs counts
identically across the resume point. Run sailormoon with
--collapse-user, truncate the output at the midpoint, resume, and
assert the result is identical to an uninterrupted run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 6634de51e6)
