Files

Benjamin Mako Hill 8bd6c2e129 add a CHANGELOG

Two version tags existed with no record of what was in them. Since
neither was ever released, the first entry describes 0.3.0 against what
people are actually running rather than against a previous release.

The redirect column change leads, because it is the one thing here that
will break code someone else has written.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

2026-08-13 16:53:52 -07:00

3.0 KiB

Raw Blame History

Changelog

0.3.0

Changes relative to what people have been running. Earlier version tags in this repository were internal markers rather than releases, so there is no previous release to compare against.

Incompatible changes

Redirect detection is now per revision. The page-level redirect_target column has been replaced by revision_is_redirect and revision_redirect_target, which classify each revision from its own text. The old column reported a page's redirect status as of the time the dump was exported and applied that single value to every revision in the page's history, which is wrong for any page that was turned into a redirect, or turned back, at some point in its life. Revisions with deleted text report null in both columns rather than false. Wikis using a localized redirect keyword can add it with --redirect-aliases; #REDIRECT is always recognized.

Code reading redirect_target needs updating.
Parquet output has been removed, along with the --partition-namespaces and --max-revisions-per-file options that only applied to it. Use JSONL, which supports --resume.
pyspark is no longer a dependency and the wikiq-spark entry point is gone. The second pass of that pipeline was never in this repository.
Python 3.11 or later is required. Python 3.9 reached end of life in October 2025, and the floor unlocks current pyarrow, more-itertools and pandas, along with the CPython 3.11 speedups on the CPU-bound work wikiq does.
The project is now called wikiq, rather than mediawiki_dump_tools.

Installation

wikiq installs with plain pip and needs no C++ compiler. Previously every install required pywikidiff2, built from an ssh-only git URL, so anyone outside the collective could not install the tool at all. It is now optional and needed only for --diff and -p wikidiff2.
-p legacy needs the new legacy extra. It was the only user of mediawiki-utilities, a 2015 package that also pulls in a MySQL client and six other packages. Everyone was paying for that to support one flag.
Dependencies resolve from PyPI rather than from pinned forks under personal accounts. Requires mwxml >= 0.3.8, which fixes eleven upstream issues; note that mwxml 0.3.7 changed namespace handling to trust the dump's embedded <ns> tag.

Added

--version, and the version is printed to stderr on every run so it lands in job logs beside the output.
Licensing: wikiq is GPL-3.0-or-later. See COPYING.

Fixed

Regex matching (-RP/-CP) crashed with a TypeError on revisions whose text or comment was deleted or suppressed. Deleted content now reports null.
Page titles in non-main namespaces were prefixed twice under mwxml 0.3.7 and later, producing titles like Category:Category:Alaska.
The test suite had 16 failing tests, and pytest test/ silently collected only half of it because the main test file matched neither of pytest's discovery patterns. It is now test_wikiq.py and the suite runs green.

3.0 KiB Raw Blame History