Prepare wikiq for release: fixes, dependency cleanup, and packaging #2

mako · 2026-08-14T00:57:46Z

mako commented

2026-08-14 00:57:46 +00:00

This is the cleanup and release-preparation work, with features kept out
so they can be reviewed separately (see #3 and #4). 33 commits, ordered
so they read forward: bugs first, then test repairs, removals, dependency
work, documentation, and packaging.

The suite passes at 65 passed, 2 skipped. The two skips are the diff
matcher tests that need an uncompressed ikwiki.xml, which is not in the
repository — that is unchanged from before.

changes to redirects (the only thing that will break existing code)

redirect_target is gone, replaced by revision_is_redirect and
revision_redirect_target.

The old column carried a page's redirect status as of export time and
applied that one value to every revision in the page's history. For any
page that became a redirect, or stopped being one, most rows were wrong.
The new columns classify each revision from its own text. Revisions with
deleted text report null rather than false.

Anything reading redirect_target needs updating. Wikis with a localized
redirect keyword can pass --redirect-aliases; #REDIRECT is always
recognized.

Also removed, as discussed: parquet output along with
--partition-namespaces and --max-revisions-per-file, and the pyspark
dependency with the wikiq-spark entry point. The project and
distribution are both renamed to wikiq.

changes to installation

wikiq now installs with plain pip and needs no C++ compiler. Previously
every install pulled pywikidiff2 from an ssh-only git URL, so nobody
outside the collective could install the tool at all.

pywikidiff2 is now imported inside the two code paths that use it —
--diff and -p wikidiff2 — so everything else works without it. It was
a single module-level import that made the whole tool unusable,
including wikiq --help.

mediawiki-utilities moved to a legacy extra for the same reason: it
backed only -p legacy, and being a 2015 package it dragged pymysql
and six others into every install.

Remaining dependencies resolve from PyPI rather than pinned forks under
personal accounts. uv.lock is now committed, so a run can be reproduced
with the versions that produced it.

Smaller things

Python 3.11 floor. GPL-3.0-or-later with COPYING. --version, also
printed to stderr each run so it lands in job logs. A CHANGELOG. Project
metadata so a PyPI page is not blank. An sdist that no longer contains a
stray virtualenv (it was 8.9 MB; it is now 40 KB).

Two crashes fixed: regex matching died with a TypeError on revisions
with deleted text, and titles in non-main namespaces were prefixed twice
under mwxml ≥ 0.3.7, producing Category:Category:Alaska.

Wikiq_Unit_Test.py is now test_wikiq.py. It matched neither of
pytest's discovery patterns, so pytest test/ had been silently running
half the suite.

This is the cleanup and release-preparation work, with features kept out so they can be reviewed separately (see #3 and #4). 33 commits, ordered so they read forward: bugs first, then test repairs, removals, dependency work, documentation, and packaging. The suite passes at 65 passed, 2 skipped. The two skips are the diff matcher tests that need an uncompressed `ikwiki.xml`, which is not in the repository — that is unchanged from before. #### changes to redirects (the only thing that will break existing code) **`redirect_target` is gone**, replaced by `revision_is_redirect` and `revision_redirect_target`. The old column carried a page's redirect status *as of export time* and applied that one value to every revision in the page's history. For any page that became a redirect, or stopped being one, most rows were wrong. The new columns classify each revision from its own text. Revisions with deleted text report null rather than false. Anything reading `redirect_target` needs updating. Wikis with a localized redirect keyword can pass `--redirect-aliases`; `#REDIRECT` is always recognized. Also removed, as discussed: parquet output along with `--partition-namespaces` and `--max-revisions-per-file`, and the pyspark dependency with the `wikiq-spark` entry point. The project and distribution are both renamed to wikiq. #### changes to installation wikiq now installs with plain `pip` and needs no C++ compiler. Previously every install pulled pywikidiff2 from an ssh-only git URL, so nobody outside the collective could install the tool at all. pywikidiff2 is now imported inside the two code paths that use it — `--diff` and `-p wikidiff2` — so everything else works without it. It was a single module-level `import` that made the whole tool unusable, including `wikiq --help`. `mediawiki-utilities` moved to a `legacy` extra for the same reason: it backed only `-p legacy`, and being a 2015 package it dragged `pymysql` and six others into every install. Remaining dependencies resolve from PyPI rather than pinned forks under personal accounts. `uv.lock` is now committed, so a run can be reproduced with the versions that produced it. #### Smaller things Python 3.11 floor. GPL-3.0-or-later with `COPYING`. `--version`, also printed to stderr each run so it lands in job logs. A CHANGELOG. Project metadata so a PyPI page is not blank. An sdist that no longer contains a stray virtualenv (it was 8.9 MB; it is now 40 KB). Two crashes fixed: regex matching died with a `TypeError` on revisions with deleted text, and titles in non-main namespaces were prefixed twice under mwxml ≥ 0.3.7, producing `Category:Category:Alaska`. `Wikiq_Unit_Test.py` is now `test_wikiq.py`. It matched neither of pytest's discovery patterns, so `pytest test/` had been silently running half the suite.

mako added 33 commits 2026-08-14 00:57:46 +00:00

fix regex matching on revisions with deleted text or comments c61eb10e47

RegexPair.matchmake crashed with a TypeError when a capture-group
pattern was applied to a revision whose text or comment was deleted or
suppressed (content is None). Guard both matching paths against None,
and make the no-capture-group path emit a None column for such
revisions instead of omitting the key entirely.

This carries forward the fix Kaylea Champion and Mako Hill made on the
mako_changes-20230429 branch (7e6cd5b), which predated the rewrite.

Adds a unit test for matchmake(None) and an end-to-end test against the
ikwiki dump, which contains revisions with deleted text and comments.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit dbcae5c64e)

only prefix namespace names onto titles that lack them 97e21e3634

WikiqPage unconditionally prepended the namespace name to titles of
pages outside the main namespace. mwxml >= 0.3.7 keeps the namespace
prefix on the title when the dump carries <ns> tags (it previously
stripped it), which made wikiq emit double-prefixed titles like
"Category:Category:Alaska". Check for the prefix before adding it.
mwxml still strips the prefix on its fallback path for dumps without
<ns> tags, so those titles are still prefixed as before.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit ec09641545)

fix test helper for build_table's three-value return a26abe0e68

build_table returns (table, reverts_column, wikitext_parser) but the
jsonl test helper still unpacked two values, so test_jsonl_noargs and
test_jsonl_tsv_equivalence failed with a ValueError before reading any
output. Also update the docstring, which still described the two-value
return.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 7c27942cb0)

regenerate test baselines for the redirect_target column 1b3093dec0

Commit 59fea19 added a redirect_target column to wikiq output but did
not regenerate the test baselines, leaving 14 baseline-comparison tests
failing. Regenerate the affected baselines from current output. Each
regenerated file was verified to differ from its old baseline only by
the addition of the new column: row counts and all values in shared
columns are identical.

Also add the noargs_sailormoon.jsonl baseline used by test_jsonl_noargs,
which was never committed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 791be0aa56)

add test for --resume with --collapse-user 7fbf62e5e0

Resume with collapsed revision groups was untested: the resume point is
the (articleid, revid) of the last written row, which for collapsed
output is the last revision of a group, and nothing verified that the
replay reconstructs group boundaries and collapsed_revs counts
identically across the resume point. Run sailormoon with
--collapse-user, truncate the output at the midpoint, resume, and
assert the result is identical to an uninterrupted run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 6634de51e6)

stop writing debug files from the diff matcher test helper f4f0ed72a0

assert_equal_enough wrote its inputs to files named "token" and "rev"
in the current directory on every invocation, littering the repository
root whenever the test suite ran.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 45f0680200)

remove orphaned regextest.tsv baseline 04ccea4848

No test reads this file; the regex tests use the basic_regextest and
capturegroup_regextest baselines.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 5fb7b5596c)

document the requirements of the two skipped diff matcher tests caf604e43d

test_diff_consistency and test_benchmark_diff both read an uncompressed
test/dumps/ikwiki.xml that is not in the repository, so enabling them
requires decompressing the .bz2 first; test_diff_consistency also
writes debug files to the current directory. Say so where the skip
markers are.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit fde1452666)

remove parquet output support 015d4f9164

Nate has given up on getting wikiq to output parquet directly because it
makes too many unpredictable large memory allocations. The workflow now
outputs JSONL in a single pass and then uses spark (wikiq_spark) to index
it as parquet in a second pass.

Remove the parquet output path along with the machinery that existed only
to support it: checkpoint files, resume temp-file merging, namespace
partitioning, and file rotation (--partition-namespaces,
--max-revisions-per-file). Resume support remains for JSONL output, where
the resume point is derived from the last complete line of the output
file. Also remove the parquet tests and baseline files.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 72a8851373)

remove pyspark dependency and wikiq_spark packaging references 6013e8c4e9

The spark indexing pass is moving out of this repository, and
src/wikiq_spark was never committed here: pyproject.toml declared a
wikiq-spark console script and wheel package that do not exist in the
tree, and pulled in pyspark for every install of a tool that never
imports it. The --print-schema flag and its Spark-format schema
converters remain, since they are pure Python and produce the schema
the external indexing pass consumes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 392211af17)

make pywikidiff2 an optional dependency 5725c66040

pywikidiff2 was imported at module scope, so every invocation of wikiq
required it -- including `wikiq --help`. It is not on PyPI, compiles a
C++ extension, and needs libthai, which made a compiler a hard
requirement for installing a tool that mostly does not need one.

Only --diff and -p wikidiff2 actually use it. Import it inside those two
code paths instead, and report what to install when it is missing rather
than failing with an ImportError traceback. The check also runs once at
startup, since both use sites sit deep in the per-revision loop and a run
can stream for hours before reaching them.

Dropping the dependency also removes the PEP 508 direct reference from
the package metadata, and with it the need for hatchling's
allow-direct-references. PyPI rejects uploads whose metadata contains a
direct URL, so this is a prerequisite for publishing wikiq there.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 12325afd46670e949abf1c0e14799e212d7ff6ce)

make mediawiki-utilities an optional dependency 70b0ebcf0c

-p legacy is the only thing that uses mediawiki-utilities, and it exists
solely to reproduce numbers from research predating the mwxml and
mwtypes split. The package is a 2015 monolith that also contains
mw.database and mw.api, so it declares requests and pymysql
unconditionally: every wikiq install pulled a MySQL client and six other
packages to support one flag, building them from an sdist that has no
wheel.

Move it to a `legacy` extra. mw.lib.persistence imports nothing from
mw.database or mw.api, so nothing about -p legacy changes for people who
install the extra, and everyone else stops paying for it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 66dcc6d8cea620a619fc19121b5c27a085b11991)

skip tests whose optional dependency is not installed b10a7d0130

With pywikidiff2 and mediawiki-utilities no longer installed by default,
the tests covering --diff, -p wikidiff2, and -p legacy cannot run on a
base install. Mark them so a base install reports skips rather than
failures, and keep the markers in wikiq_test_utils.py so all three test
modules share one definition of what each feature needs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 59d5a9d46a04320bad5d81e0edf5ecd11f9c2a9b)

resolve dependencies from PyPI instead of pinned git forks ac8110894b

Drop the [tool.uv.sources] pins of deltas, mwxml, and yamlconf to forks
under a personal github account:

- The deltas fork was byte-identical to upstream.
- The mwxml fork carried one patch to mwxml.map(), which wikiq does not
  call, and was missing eleven upstream commits of fixes. Require
  mwxml >= 0.3.8 for those fixes; note that 0.3.7 changed namespace
  handling to trust the dump's embedded <ns> tag rather than overriding
  it based on the title prefix.
- The yamlconf fork loosened an old pyyaml pin. yamlconf is not imported
  by wikiq and is only needed transitively via deltas, so drop the direct
  dependency. PyPI's 0.2.6 pinned pyyaml == 5.4.1, which has no wheels
  for 3.11+ and no longer builds from source; that was fixed upstream in
  0.2.7, and the project has since moved to the mediawiki-utilities
  organization.

With pywikidiff2 no longer a dependency at all, no uv-specific
configuration remains and the package installs with plain pip.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

require Python 3.11 15379c1c1a

Python 3.9 reached end of life in October 2025, and 3.11 is where the
current ecosystem sits: it unlocks pyarrow 25 (we were held at 21, the
last release supporting 3.9), more-itertools 11, and pandas 3 for the
test suite, along with the large CPython 3.11 interpreter speedups on
exactly the kind of CPU-bound work wikiq does. Nothing in the code
needed changes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

convert README to markdown c31833e064

Replace the auto-converted README.md (which carried pandoc title-ref
spans and a garbled Tests heading from a malformed rst underline) with
a clean conversion of README.rst, and remove the rst version.
pyproject.toml already declares README.md as the package readme.
Content is unchanged; bringing it up to date is left for a separate
change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit fa8adc30ca)

license under the GNU GPL, version 3 or later ceb230904e

Add the full license text as COPYING (the same text Debian ships in
common-licenses), declare the license and license file in
pyproject.toml, add copyright and permission notices to the source
files, and describe the license in the README. Also replace the
placeholder package description in pyproject.toml.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 90a14cf999)

update README for the current tool d4111dc4f3

Describe what wikiq is and rewrite the installation, usage, and test
instructions to match the current code: pip or uv installation from the
gitea repository, the Python 3.11 requirement, output format selection
by extension, and the current option set including persistence methods,
wikitext extraction, regex matching and counting, and JSONL resume. Add
an authors section crediting the Community Data Science Collective
contributors and the earlier Python and C++ versions of the tool.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit d6d0aab604)

make redirect detection revision-level b8dddf6af8

The redirect_target column added in 59fea19 was page-level: the page's
state at dump time, stamped onto every revision row of the page. That
invited the wrong inference that a given revision was a redirect, and
the column was empty for dumps whose export never wrote <redirect>
elements, such as the 2023 wikitravel.org scrapes; a Swedish Wikitravel
scrape with 732 in-text redirect revisions produced no redirect signal
at all.

Remove that column and detect redirects from each revision's own text
instead. revision_is_redirect records whether the text begins with a
redirect directive and revision_redirect_target records the directive's
link target with any fragment and label stripped. #REDIRECT is
recognized on every wiki; localized keywords (e.g. OMDIRIGERING on
Swedish wikis) can be added with --redirect-aliases. Revisions with
deleted or unavailable text get nulls in both columns. The redirect-map
use case behind 59fea19 survives: the last revision's
revision_redirect_target per page reconstructs the page-level map, now
also on dumps without <redirect> elements.

Document that title and namespace are page-level identity values as of
the time of export, not historical facts about each revision.

Regenerate the test baselines for the column change. Every regenerated
file was verified to differ from its predecessor only by removing
redirect_target and adding the two new columns, with identical values
in all shared columns.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 40132f04dd)

rename Wikiq_Unit_Test.py to test_wikiq.py af8c45e8c0

The file was the only one in test/ using mixed case and a _Test suffix,
and pytest discovers test_*.py and *_test.py case sensitively, so it
matched neither pattern. `pytest test/` -- the command the README
documents -- quietly collected only test_resume.py and
test_wiki_diff_matcher.py, meaning the 36 tests here, including every
baseline comparison, ran only when the file was named explicitly on the
command line.

Renaming fixes discovery and matches the other test modules, rather than
teaching pytest an extra pattern to accommodate one odd name.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 5a8c08e62cba5593c4b53d83f99ef389e3fd3f94)

ignore the venv and local tool state fbaafd1381

On klone the venv is created inside the repo with create_cdsc_venv, so
it otherwise shows up as untracked in every git status. Claude Code
keeps its per-project permission list in .claude/settings.local.json,
which is personal to whoever is running it, and writes it atomically via
sibling temp files, so the pattern covers those too. Only that file is
ignored, leaving room to commit a shared .claude/settings.json later.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

remove the stale .gitmodules a68a596bf4

It declares a mediawiki-php-wikidiff2 submodule, but there is no gitlink
for it in the index and no such directory on disk, so `git submodule`
has nothing to act on. Leftover from 5a3e410, when wikidiff2 persistence
was first being tried; the wikidiff2 source now reaches wikiq through
pywikidiff2 instead.

It was also being swept into the source distribution.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 2024dcc97d4013f0b5fc9ac89bf44c5ed8a2c06d)

keep local files out of the source distribution 421e9c0739

With no sdist target configured, hatchling walked the whole working tree
and included whatever it found. The resulting 8.9MB sdist carried 448
files from a virtualenv left in the repository root, plus .claude/ and
.gitmodules, while omitting the test dumps -- those are gitignored, so
the tests it did ship could not have run.

List the sdist contents explicitly. It is now 40KB of source, README and
COPYING, and both it and the wheel pass twine check. Tests run from a
clone rather than from the sdist, so test/ is deliberately excluded.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 28406563bcdb7b2f31412e511ea30f66eb3620b5)

add the project metadata a published package needs 1125194093

The package declared no authors, no URLs, no classifiers and no
keywords, so a PyPI page for it would be blank and it would not turn up
in a search. Fill those in.

Contributors are listed by name only. They are the same people the
README credits, so this discloses nothing new, and publishing other
people's email addresses to PyPI is not ours to decide.

Also mirror the dev dependencies into [project.optional-dependencies].
[dependency-groups] is PEP 735, which only uv reads, so there was no way
for a pip user to install what the tests need; `pip install -e '.[dev]'`
now works. The two lists have to be kept in step by hand.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 7b3c0f9a80e39990df90124a7c592a80d93e0d91)

correct the test suite runtime in the README 507cc24328

The suite completes in about ninety seconds, not fifteen minutes. The
old figure discouraged running it as part of an ordinary change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 2bee25488b)

document the optional dependencies ff1f42dae5

Explain in the installation section that a base install needs no
compiler, and give the install command for each optional feature. The
tests section said to run the suite with `uv run pytest`, which
contradicted the installation section's statement that uv is not
required; use plain pytest and note that tests for the optional features
skip when their dependency is absent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 1c4c8dfe31cb8a8eea0dc72b7ca136b44e134c9f)

run the whole suite from runtest.sh 77f185888f

It invoked pytest on test_wiki_diff_matcher.py alone, so it exercised 19
of the 69 tests despite its name, and it required uv, which contradicts
the README's statement that uv is optional.

Point it at test/ and use whichever python is on the path. It cds to the
repository root first, since several tests open fixture files by paths
relative to it, and passes any extra arguments through.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 7fff65c23d7c7a2868abccb235ec376fe093d4c1)

add --version and set the version to 0.3.0 b073b038af

wikiq had no way to report which version produced a dataset, which
matters when output columns change between runs that end up in the same
analysis. Add a --version flag, and print the version to stderr on every
run so it lands in job logs alongside the output.

0.3.0 is the first release published anywhere. The earlier 0.1.1 and
0.2.0 tags were internal markers that were never released, so the
version numbering starts being meaningful to users here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

rename the project from mediawiki_dump_tools to wikiq 1b55acb01f

wikiq is what the tool is called, what people type, and what they would
search for; mediawiki_dump_tools was only ever the name of the directory
it lived in. The repository is being renamed on gitea and github to
match, so rename the distribution with it.

This also settles the name to claim on PyPI. The metadata lookup that
backs --version takes the distribution name as a literal, so it changes
here too, along with the install command wikiq prints when -p legacy is
used without its extra.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

put the base install ahead of the optional features in the README 0fa8b3d9ba

The installation section led with the two optional dependencies and left
the decompression tools until last, which put the awkward parts first
and buried something every run actually needs. Describe the base install
and the compression handling together, then the optional features under
their own headings.

Drop the note about uv. It said only that uv also works, which anyone
who uses uv already knows.

Also open with what wikiq is rather than describing the repository as a
collection of tools, now that the repository is named after the program.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

add a CHANGELOG 8bd6c2e129

Two version tags existed with no record of what was in them. Since
neither was ever released, the first entry describes 0.3.0 against what
people are actually running rather than against a previous release.

The redirect column change leads, because it is the one thing here that
will break code someone else has written.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

sort the imports in wikiq/__init__.py 268c212469

Four imports sat below the module constants, where pathlib and pyarrow
had drifted after some earlier edit, and the local wikiq imports were
mixed in with third-party ones. Group them as standard library,
third-party, then local, each alphabetized, and put the version lookup
and the constants after all of them.

Pure movement: no import added or removed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

commit the uv lockfile 57988849cc

wikiq produces datasets that get analyzed for years, and reproducing a
run means reproducing the dependency versions that produced it. Without
a lockfile in the repository there is no record of what a given output
was actually generated with, and the loose lower bounds in pyproject.toml
will resolve differently over time.

The lockfile pins all 72 packages in the resolution, including the
transitive ones from the mediawiki-utilities stack that are old enough to
be worth pinning explicitly.

This does not constrain anyone installing with pip, which ignores the
lockfile; it records what `uv sync` resolves so that a run can be
reproduced deliberately.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin release_review:release_review

git checkout release_review

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: collective/wikiq#2