Describe what wikiq is and rewrite the installation, usage, and test
instructions to match the current code: pip or uv installation from the
gitea repository, the Python 3.11 requirement, output format selection
by extension, and the current option set including persistence methods,
wikitext extraction, regex matching and counting, and JSONL resume. Add
an authors section crediting the Community Data Science Collective
contributors and the earlier Python and C++ versions of the tool.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit d6d0aab604)
Add the full license text as COPYING (the same text Debian ships in
common-licenses), declare the license and license file in
pyproject.toml, add copyright and permission notices to the source
files, and describe the license in the README. Also replace the
placeholder package description in pyproject.toml.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 90a14cf999)
Replace the auto-converted README.md (which carried pandoc title-ref
spans and a garbled Tests heading from a malformed rst underline) with
a clean conversion of README.rst, and remove the rst version.
pyproject.toml already declares README.md as the package readme.
Content is unchanged; bringing it up to date is left for a separate
change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit fa8adc30ca)
Python 3.9 reached end of life in October 2025, and 3.11 is where the
current ecosystem sits: it unlocks pyarrow 25 (we were held at 21, the
last release supporting 3.9), more-itertools 11, and pandas 3 for the
test suite, along with the large CPython 3.11 interpreter speedups on
exactly the kind of CPU-bound work wikiq does. Nothing in the code
needed changes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Drop the [tool.uv.sources] pins of deltas, mwxml, and yamlconf to forks
under a personal github account:
- The deltas fork was byte-identical to upstream.
- The mwxml fork carried one patch to mwxml.map(), which wikiq does not
call, and was missing eleven upstream commits of fixes. Require
mwxml >= 0.3.8 for those fixes; note that 0.3.7 changed namespace
handling to trust the dump's embedded <ns> tag rather than overriding
it based on the title prefix.
- The yamlconf fork loosened an old pyyaml pin. yamlconf is not imported
by wikiq and is only needed transitively via deltas, so drop the direct
dependency. PyPI's 0.2.6 pinned pyyaml == 5.4.1, which has no wheels
for 3.11+ and no longer builds from source; that was fixed upstream in
0.2.7, and the project has since moved to the mediawiki-utilities
organization.
With pywikidiff2 no longer a dependency at all, no uv-specific
configuration remains and the package installs with plain pip.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
With pywikidiff2 and mediawiki-utilities no longer installed by default,
the tests covering --diff, -p wikidiff2, and -p legacy cannot run on a
base install. Mark them so a base install reports skips rather than
failures, and keep the markers in wikiq_test_utils.py so all three test
modules share one definition of what each feature needs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 59d5a9d46a04320bad5d81e0edf5ecd11f9c2a9b)
-p legacy is the only thing that uses mediawiki-utilities, and it exists
solely to reproduce numbers from research predating the mwxml and
mwtypes split. The package is a 2015 monolith that also contains
mw.database and mw.api, so it declares requests and pymysql
unconditionally: every wikiq install pulled a MySQL client and six other
packages to support one flag, building them from an sdist that has no
wheel.
Move it to a `legacy` extra. mw.lib.persistence imports nothing from
mw.database or mw.api, so nothing about -p legacy changes for people who
install the extra, and everyone else stops paying for it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 66dcc6d8cea620a619fc19121b5c27a085b11991)
pywikidiff2 was imported at module scope, so every invocation of wikiq
required it -- including `wikiq --help`. It is not on PyPI, compiles a
C++ extension, and needs libthai, which made a compiler a hard
requirement for installing a tool that mostly does not need one.
Only --diff and -p wikidiff2 actually use it. Import it inside those two
code paths instead, and report what to install when it is missing rather
than failing with an ImportError traceback. The check also runs once at
startup, since both use sites sit deep in the per-revision loop and a run
can stream for hours before reaching them.
Dropping the dependency also removes the PEP 508 direct reference from
the package metadata, and with it the need for hatchling's
allow-direct-references. PyPI rejects uploads whose metadata contains a
direct URL, so this is a prerequisite for publishing wikiq there.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 12325afd46670e949abf1c0e14799e212d7ff6ce)
The spark indexing pass is moving out of this repository, and
src/wikiq_spark was never committed here: pyproject.toml declared a
wikiq-spark console script and wheel package that do not exist in the
tree, and pulled in pyspark for every install of a tool that never
imports it. The --print-schema flag and its Spark-format schema
converters remain, since they are pure Python and produce the schema
the external indexing pass consumes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 392211af17)
Nate has given up on getting wikiq to output parquet directly because it
makes too many unpredictable large memory allocations. The workflow now
outputs JSONL in a single pass and then uses spark (wikiq_spark) to index
it as parquet in a second pass.
Remove the parquet output path along with the machinery that existed only
to support it: checkpoint files, resume temp-file merging, namespace
partitioning, and file rotation (--partition-namespaces,
--max-revisions-per-file). Resume support remains for JSONL output, where
the resume point is derived from the last complete line of the output
file. Also remove the parquet tests and baseline files.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 72a8851373)
test_diff_consistency and test_benchmark_diff both read an uncompressed
test/dumps/ikwiki.xml that is not in the repository, so enabling them
requires decompressing the .bz2 first; test_diff_consistency also
writes debug files to the current directory. Say so where the skip
markers are.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit fde1452666)
No test reads this file; the regex tests use the basic_regextest and
capturegroup_regextest baselines.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 5fb7b5596c)
assert_equal_enough wrote its inputs to files named "token" and "rev"
in the current directory on every invocation, littering the repository
root whenever the test suite ran.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 45f0680200)
Resume with collapsed revision groups was untested: the resume point is
the (articleid, revid) of the last written row, which for collapsed
output is the last revision of a group, and nothing verified that the
replay reconstructs group boundaries and collapsed_revs counts
identically across the resume point. Run sailormoon with
--collapse-user, truncate the output at the midpoint, resume, and
assert the result is identical to an uninterrupted run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 6634de51e6)
Commit 59fea19 added a redirect_target column to wikiq output but did
not regenerate the test baselines, leaving 14 baseline-comparison tests
failing. Regenerate the affected baselines from current output. Each
regenerated file was verified to differ from its old baseline only by
the addition of the new column: row counts and all values in shared
columns are identical.
Also add the noargs_sailormoon.jsonl baseline used by test_jsonl_noargs,
which was never committed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 791be0aa56)
build_table returns (table, reverts_column, wikitext_parser) but the
jsonl test helper still unpacked two values, so test_jsonl_noargs and
test_jsonl_tsv_equivalence failed with a ValueError before reading any
output. Also update the docstring, which still described the two-value
return.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 7c27942cb0)
WikiqPage unconditionally prepended the namespace name to titles of
pages outside the main namespace. mwxml >= 0.3.7 keeps the namespace
prefix on the title when the dump carries <ns> tags (it previously
stripped it), which made wikiq emit double-prefixed titles like
"Category:Category:Alaska". Check for the prefix before adding it.
mwxml still strips the prefix on its fallback path for dumps without
<ns> tags, so those titles are still prefixed as before.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit ec09641545)
RegexPair.matchmake crashed with a TypeError when a capture-group
pattern was applied to a revision whose text or comment was deleted or
suppressed (content is None). Guard both matching paths against None,
and make the no-capture-group path emit a None column for such
revisions instead of omitting the key entirely.
This carries forward the fix Kaylea Champion and Mako Hill made on the
mako_changes-20230429 branch (7e6cd5b), which predated the rewrite.
Adds a unit test for matchmake(None) and an end-to-end test against the
ikwiki dump, which contains revisions with deleted text and comments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit dbcae5c64e)
Per-page adaptive parse skip: once mwparserfromhell times out on a
revision of size N bytes, force-timeout future revisions on the same
page whose text is >= N bytes instead of waiting another 60s on each
one. State resets per page, so well-behaved pages are unaffected.
Motivated by zhwiki bot-vandalism log pages (e.g. ns=4
'Wikipedia:当前的破坏/...' at 4.6GB across 6133 revisions) where every
revision triggers the 60s SIGALRM, blocking the worker for hours.
- Implement per-namespace resume points (dict mapping namespace -> (pageid, revid))
to correctly handle interleaved dump ordering in partitioned output
- Extract resume functionality to dedicated resume.py module
- Add graceful shutdown handling via shutdown_requested flag (CLI-level only)
- Use lazy ParquetWriter creation to avoid empty files on early exit
- Refactor writing logic to _write_batch() helper method
- Simplify control flow by replacing continue statements with should_write flag
