The installation section led with the two optional dependencies and left
the decompression tools until last, which put the awkward parts first
and buried something every run actually needs. Describe the base install
and the compression handling together, then the optional features under
their own headings.
Drop the note about uv. It said only that uv also works, which anyone
who uses uv already knows.
Also open with what wikiq is rather than describing the repository as a
collection of tools, now that the repository is named after the program.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
wikiq is what the tool is called, what people type, and what they would
search for; mediawiki_dump_tools was only ever the name of the directory
it lived in. The repository is being renamed on gitea and github to
match, so rename the distribution with it.
This also settles the name to claim on PyPI. The metadata lookup that
backs --version takes the distribution name as a literal, so it changes
here too, along with the install command wikiq prints when -p legacy is
used without its extra.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
wikiq had no way to report which version produced a dataset, which
matters when output columns change between runs that end up in the same
analysis. Add a --version flag, and print the version to stderr on every
run so it lands in job logs alongside the output.
0.3.0 is the first release published anywhere. The earlier 0.1.1 and
0.2.0 tags were internal markers that were never released, so the
version numbering starts being meaningful to users here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It invoked pytest on test_wiki_diff_matcher.py alone, so it exercised 19
of the 69 tests despite its name, and it required uv, which contradicts
the README's statement that uv is optional.
Point it at test/ and use whichever python is on the path. It cds to the
repository root first, since several tests open fixture files by paths
relative to it, and passes any extra arguments through.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 7fff65c23d7c7a2868abccb235ec376fe093d4c1)
Explain in the installation section that a base install needs no
compiler, and give the install command for each optional feature. The
tests section said to run the suite with `uv run pytest`, which
contradicted the installation section's statement that uv is not
required; use plain pytest and note that tests for the optional features
skip when their dependency is absent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 1c4c8dfe31cb8a8eea0dc72b7ca136b44e134c9f)
The suite completes in about ninety seconds, not fifteen minutes. The
old figure discouraged running it as part of an ordinary change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 2bee25488b)
The package declared no authors, no URLs, no classifiers and no
keywords, so a PyPI page for it would be blank and it would not turn up
in a search. Fill those in.
Contributors are listed by name only. They are the same people the
README credits, so this discloses nothing new, and publishing other
people's email addresses to PyPI is not ours to decide.
Also mirror the dev dependencies into [project.optional-dependencies].
[dependency-groups] is PEP 735, which only uv reads, so there was no way
for a pip user to install what the tests need; `pip install -e '.[dev]'`
now works. The two lists have to be kept in step by hand.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 7b3c0f9a80e39990df90124a7c592a80d93e0d91)
With no sdist target configured, hatchling walked the whole working tree
and included whatever it found. The resulting 8.9MB sdist carried 448
files from a virtualenv left in the repository root, plus .claude/ and
.gitmodules, while omitting the test dumps -- those are gitignored, so
the tests it did ship could not have run.
List the sdist contents explicitly. It is now 40KB of source, README and
COPYING, and both it and the wheel pass twine check. Tests run from a
clone rather than from the sdist, so test/ is deliberately excluded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 28406563bcdb7b2f31412e511ea30f66eb3620b5)
It declares a mediawiki-php-wikidiff2 submodule, but there is no gitlink
for it in the index and no such directory on disk, so `git submodule`
has nothing to act on. Leftover from 5a3e410, when wikidiff2 persistence
was first being tried; the wikidiff2 source now reaches wikiq through
pywikidiff2 instead.
It was also being swept into the source distribution.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 2024dcc97d4013f0b5fc9ac89bf44c5ed8a2c06d)
On klone the venv is created inside the repo with create_cdsc_venv, so
it otherwise shows up as untracked in every git status. Claude Code
keeps its per-project permission list in .claude/settings.local.json,
which is personal to whoever is running it, and writes it atomically via
sibling temp files, so the pattern covers those too. Only that file is
ignored, leaving room to commit a shared .claude/settings.json later.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The file was the only one in test/ using mixed case and a _Test suffix,
and pytest discovers test_*.py and *_test.py case sensitively, so it
matched neither pattern. `pytest test/` -- the command the README
documents -- quietly collected only test_resume.py and
test_wiki_diff_matcher.py, meaning the 36 tests here, including every
baseline comparison, ran only when the file was named explicitly on the
command line.
Renaming fixes discovery and matches the other test modules, rather than
teaching pytest an extra pattern to accommodate one odd name.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 5a8c08e62cba5593c4b53d83f99ef389e3fd3f94)
The redirect_target column added in 59fea19 was page-level: the page's
state at dump time, stamped onto every revision row of the page. That
invited the wrong inference that a given revision was a redirect, and
the column was empty for dumps whose export never wrote <redirect>
elements, such as the 2023 wikitravel.org scrapes; a Swedish Wikitravel
scrape with 732 in-text redirect revisions produced no redirect signal
at all.
Remove that column and detect redirects from each revision's own text
instead. revision_is_redirect records whether the text begins with a
redirect directive and revision_redirect_target records the directive's
link target with any fragment and label stripped. #REDIRECT is
recognized on every wiki; localized keywords (e.g. OMDIRIGERING on
Swedish wikis) can be added with --redirect-aliases. Revisions with
deleted or unavailable text get nulls in both columns. The redirect-map
use case behind 59fea19 survives: the last revision's
revision_redirect_target per page reconstructs the page-level map, now
also on dumps without <redirect> elements.
Document that title and namespace are page-level identity values as of
the time of export, not historical facts about each revision.
Regenerate the test baselines for the column change. Every regenerated
file was verified to differ from its predecessor only by removing
redirect_target and adding the two new columns, with identical values
in all shared columns.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 40132f04dd)
Describe what wikiq is and rewrite the installation, usage, and test
instructions to match the current code: pip or uv installation from the
gitea repository, the Python 3.11 requirement, output format selection
by extension, and the current option set including persistence methods,
wikitext extraction, regex matching and counting, and JSONL resume. Add
an authors section crediting the Community Data Science Collective
contributors and the earlier Python and C++ versions of the tool.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit d6d0aab604)
Add the full license text as COPYING (the same text Debian ships in
common-licenses), declare the license and license file in
pyproject.toml, add copyright and permission notices to the source
files, and describe the license in the README. Also replace the
placeholder package description in pyproject.toml.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 90a14cf999)
Replace the auto-converted README.md (which carried pandoc title-ref
spans and a garbled Tests heading from a malformed rst underline) with
a clean conversion of README.rst, and remove the rst version.
pyproject.toml already declares README.md as the package readme.
Content is unchanged; bringing it up to date is left for a separate
change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit fa8adc30ca)
Python 3.9 reached end of life in October 2025, and 3.11 is where the
current ecosystem sits: it unlocks pyarrow 25 (we were held at 21, the
last release supporting 3.9), more-itertools 11, and pandas 3 for the
test suite, along with the large CPython 3.11 interpreter speedups on
exactly the kind of CPU-bound work wikiq does. Nothing in the code
needed changes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Drop the [tool.uv.sources] pins of deltas, mwxml, and yamlconf to forks
under a personal github account:
- The deltas fork was byte-identical to upstream.
- The mwxml fork carried one patch to mwxml.map(), which wikiq does not
call, and was missing eleven upstream commits of fixes. Require
mwxml >= 0.3.8 for those fixes; note that 0.3.7 changed namespace
handling to trust the dump's embedded <ns> tag rather than overriding
it based on the title prefix.
- The yamlconf fork loosened an old pyyaml pin. yamlconf is not imported
by wikiq and is only needed transitively via deltas, so drop the direct
dependency. PyPI's 0.2.6 pinned pyyaml == 5.4.1, which has no wheels
for 3.11+ and no longer builds from source; that was fixed upstream in
0.2.7, and the project has since moved to the mediawiki-utilities
organization.
With pywikidiff2 no longer a dependency at all, no uv-specific
configuration remains and the package installs with plain pip.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
With pywikidiff2 and mediawiki-utilities no longer installed by default,
the tests covering --diff, -p wikidiff2, and -p legacy cannot run on a
base install. Mark them so a base install reports skips rather than
failures, and keep the markers in wikiq_test_utils.py so all three test
modules share one definition of what each feature needs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 59d5a9d46a04320bad5d81e0edf5ecd11f9c2a9b)
-p legacy is the only thing that uses mediawiki-utilities, and it exists
solely to reproduce numbers from research predating the mwxml and
mwtypes split. The package is a 2015 monolith that also contains
mw.database and mw.api, so it declares requests and pymysql
unconditionally: every wikiq install pulled a MySQL client and six other
packages to support one flag, building them from an sdist that has no
wheel.
Move it to a `legacy` extra. mw.lib.persistence imports nothing from
mw.database or mw.api, so nothing about -p legacy changes for people who
install the extra, and everyone else stops paying for it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 66dcc6d8cea620a619fc19121b5c27a085b11991)
pywikidiff2 was imported at module scope, so every invocation of wikiq
required it -- including `wikiq --help`. It is not on PyPI, compiles a
C++ extension, and needs libthai, which made a compiler a hard
requirement for installing a tool that mostly does not need one.
Only --diff and -p wikidiff2 actually use it. Import it inside those two
code paths instead, and report what to install when it is missing rather
than failing with an ImportError traceback. The check also runs once at
startup, since both use sites sit deep in the per-revision loop and a run
can stream for hours before reaching them.
Dropping the dependency also removes the PEP 508 direct reference from
the package metadata, and with it the need for hatchling's
allow-direct-references. PyPI rejects uploads whose metadata contains a
direct URL, so this is a prerequisite for publishing wikiq there.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 12325afd46670e949abf1c0e14799e212d7ff6ce)
The spark indexing pass is moving out of this repository, and
src/wikiq_spark was never committed here: pyproject.toml declared a
wikiq-spark console script and wheel package that do not exist in the
tree, and pulled in pyspark for every install of a tool that never
imports it. The --print-schema flag and its Spark-format schema
converters remain, since they are pure Python and produce the schema
the external indexing pass consumes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 392211af17)
Nate has given up on getting wikiq to output parquet directly because it
makes too many unpredictable large memory allocations. The workflow now
outputs JSONL in a single pass and then uses spark (wikiq_spark) to index
it as parquet in a second pass.
Remove the parquet output path along with the machinery that existed only
to support it: checkpoint files, resume temp-file merging, namespace
partitioning, and file rotation (--partition-namespaces,
--max-revisions-per-file). Resume support remains for JSONL output, where
the resume point is derived from the last complete line of the output
file. Also remove the parquet tests and baseline files.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 72a8851373)
test_diff_consistency and test_benchmark_diff both read an uncompressed
test/dumps/ikwiki.xml that is not in the repository, so enabling them
requires decompressing the .bz2 first; test_diff_consistency also
writes debug files to the current directory. Say so where the skip
markers are.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit fde1452666)
No test reads this file; the regex tests use the basic_regextest and
capturegroup_regextest baselines.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 5fb7b5596c)
assert_equal_enough wrote its inputs to files named "token" and "rev"
in the current directory on every invocation, littering the repository
root whenever the test suite ran.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 45f0680200)
Resume with collapsed revision groups was untested: the resume point is
the (articleid, revid) of the last written row, which for collapsed
output is the last revision of a group, and nothing verified that the
replay reconstructs group boundaries and collapsed_revs counts
identically across the resume point. Run sailormoon with
--collapse-user, truncate the output at the midpoint, resume, and
assert the result is identical to an uninterrupted run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 6634de51e6)
Commit 59fea19 added a redirect_target column to wikiq output but did
not regenerate the test baselines, leaving 14 baseline-comparison tests
failing. Regenerate the affected baselines from current output. Each
regenerated file was verified to differ from its old baseline only by
the addition of the new column: row counts and all values in shared
columns are identical.
Also add the noargs_sailormoon.jsonl baseline used by test_jsonl_noargs,
which was never committed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 791be0aa56)
build_table returns (table, reverts_column, wikitext_parser) but the
jsonl test helper still unpacked two values, so test_jsonl_noargs and
test_jsonl_tsv_equivalence failed with a ValueError before reading any
output. Also update the docstring, which still described the two-value
return.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 7c27942cb0)
WikiqPage unconditionally prepended the namespace name to titles of
pages outside the main namespace. mwxml >= 0.3.7 keeps the namespace
prefix on the title when the dump carries <ns> tags (it previously
stripped it), which made wikiq emit double-prefixed titles like
"Category:Category:Alaska". Check for the prefix before adding it.
mwxml still strips the prefix on its fallback path for dumps without
<ns> tags, so those titles are still prefixed as before.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit ec09641545)
RegexPair.matchmake crashed with a TypeError when a capture-group
pattern was applied to a revision whose text or comment was deleted or
suppressed (content is None). Guard both matching paths against None,
and make the no-capture-group path emit a None column for such
revisions instead of omitting the key entirely.
This carries forward the fix Kaylea Champion and Mako Hill made on the
mako_changes-20230429 branch (7e6cd5b), which predated the rewrite.
Adds a unit test for matchmake(None) and an end-to-end test against the
ikwiki dump, which contains revisions with deleted text and comments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit dbcae5c64e)
Per-page adaptive parse skip: once mwparserfromhell times out on a
revision of size N bytes, force-timeout future revisions on the same
page whose text is >= N bytes instead of waiting another 60s on each
one. State resets per page, so well-behaved pages are unaffected.
Motivated by zhwiki bot-vandalism log pages (e.g. ns=4
'Wikipedia:当前的破坏/...' at 4.6GB across 6133 revisions) where every
revision triggers the 60s SIGALRM, blocking the worker for hours.
