document regex match counting in the README

Describes -RPc and -CPc alongside the regex matching options they
modify. Kept with the feature rather than on the release branch, so the
README there documents only what that branch actually provides.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

.gitignore

@@ -4,12 +4,11 @@
 *.xml.xz
 *.swp
 
-# Lockfiles
-uv.lock
-
 # JetBrains
 /.idea
 
+/.claude/settings.local.json*
+
 # Python build and test output
 __pycache__/
 /test/test_output/

.gitmodules

@@ -1,3 +0,0 @@
-[submodule "mediawiki-php-wikidiff2"]
-	path = mediawiki-php-wikidiff2
-	url = https://github.com/wikimedia/mediawiki-php-wikidiff2/

CHANGELOG.md

@@ -0,0 +1,73 @@
+# Changelog
+
+## 0.3.0
+
+Changes relative to what people have been running. Earlier version tags
+in this repository were internal markers rather than releases, so there
+is no previous release to compare against.
+
+### Incompatible changes
+
+- **Redirect detection is now per revision.** The page-level
+  `redirect_target` column has been replaced by `revision_is_redirect`
+  and `revision_redirect_target`, which classify each revision from its
+  own text. The old column reported a page's redirect status as of the
+  time the dump was exported and applied that single value to every
+  revision in the page's history, which is wrong for any page that was
+  turned into a redirect, or turned back, at some point in its life.
+  Revisions with deleted text report null in both columns rather than
+  false. Wikis using a localized redirect keyword can add it with
+  `--redirect-aliases`; `#REDIRECT` is always recognized.
+
+  Code reading `redirect_target` needs updating.
+
+- **Parquet output has been removed**, along with the
+  `--partition-namespaces` and `--max-revisions-per-file` options that
+  only applied to it. Use JSONL, which supports `--resume`.
+
+- **pyspark is no longer a dependency** and the `wikiq-spark` entry point
+  is gone. The second pass of that pipeline was never in this
+  repository.
+
+- **Python 3.11 or later is required.** Python 3.9 reached end of life in
+  October 2025, and the floor unlocks current pyarrow, more-itertools and
+  pandas, along with the CPython 3.11 speedups on the CPU-bound work
+  wikiq does.
+
+- **The project is now called wikiq**, rather than mediawiki_dump_tools.
+
+### Installation
+
+- **wikiq installs with plain `pip` and needs no C++ compiler.**
+  Previously every install required pywikidiff2, built from an ssh-only
+  git URL, so anyone outside the collective could not install the tool at
+  all. It is now optional and needed only for `--diff` and
+  `-p wikidiff2`.
+
+- **`-p legacy` needs the new `legacy` extra.** It was the only user of
+  `mediawiki-utilities`, a 2015 package that also pulls in a MySQL client
+  and six other packages. Everyone was paying for that to support one
+  flag.
+
+- **Dependencies resolve from PyPI** rather than from pinned forks under
+  personal accounts. Requires mwxml >= 0.3.8, which fixes eleven upstream
+  issues; note that mwxml 0.3.7 changed namespace handling to trust the
+  dump's embedded `<ns>` tag.
+
+### Added
+
+- `--version`, and the version is printed to stderr on every run so it
+  lands in job logs beside the output.
+- Licensing: wikiq is GPL-3.0-or-later. See `COPYING`.
+
+### Fixed
+
+- Regex matching (`-RP`/`-CP`) crashed with a `TypeError` on revisions
+  whose text or comment was deleted or suppressed. Deleted content now
+  reports null.
+- Page titles in non-main namespaces were prefixed twice under mwxml
+  0.3.7 and later, producing titles like `Category:Category:Alaska`.
+- The test suite had 16 failing tests, and `pytest test/` silently
+  collected only half of it because the main test file matched neither of
+  pytest's discovery patterns. It is now `test_wikiq.py` and the suite
+  runs green.

README.md

@@ -1,33 +1,53 @@
-# mediawiki_dump_tools
+# wikiq
 
-Tools for converting MediaWiki XML database dumps—the "history" dumps
+wikiq converts MediaWiki XML database dumps—the "history" dumps that
-that include every revision of every page—into tabular datasets for
+include every revision of every page—into tabular datasets for research.
-research. The main tool is `wikiq`, a command line program that produces
+It is a command line program that produces one row per revision with
-one row per revision with metadata such as the page, namespace,
+metadata such as the page, namespace, timestamp, editor, text size, and
-timestamp, editor, text size, and revert status, plus a range of
+revert status, plus a range of optional computed columns.
-optional computed columns.
 
 ## Installation
 
 wikiq requires Python 3.11 or later. Install it with pip from a clone of
 this repository:
 
-    git clone https://gitea.communitydata.science/collective/mediawiki_dump_tools.git
+    git clone https://gitea.communitydata.science/collective/wikiq.git
-    cd mediawiki_dump_tools
+    cd wikiq
     pip install .
 
-[uv](https://docs.astral.sh/uv/) works too (`uv sync`) and is convenient
-for development, but it is not required.
-
-One dependency, `pywikidiff2`, is installed from source and compiles a
-C++ extension, so a C++ compiler must be available at install time.
-
 Wikimedia dumps are usually compressed as 7z (most common), gz, or bz2.
 wikiq reads these by running your system's decompression tools, so it
 depends on `7za`, `zcat`, and `bzcat` for those respective formats. On
 Debian or Ubuntu, `apt install 7zip` provides `7za`; the others are
 standard.
 
+That is everything most uses need, and none of it requires a compiler.
+Two further features carry heavier dependencies, so they are kept out of
+the base install.
+
+### Diffs and wikidiff2 persistence
+
+`--diff` and `-p wikidiff2` need
+[pywikidiff2](https://gitea.communitydata.science/groceryheist/pywikidiff2),
+a Python binding for MediaWiki's wikidiff2 diff engine. It is not on
+PyPI and compiles a C++ extension, so it needs a C++ compiler and
+libthai:
+
+    pip install 'pywikidiff2 @ git+https://gitea.communitydata.science/groceryheist/pywikidiff2.git'
+
+The other persistence methods, including the default `-p sequence`, do
+not need it.
+
+### Legacy persistence
+
+`-p legacy` needs `mediawiki-utilities`, which is only useful for
+reproducing results from older research projects:
+
+    pip install '.[legacy]'
+
+wikiq tells you which of these to install if you use an option that
+needs one.
+
 ## Usage
 
     wikiq dump.xml.7z -o output/
@@ -80,12 +100,16 @@ The most commonly useful options (`wikiq --help` describes them all):
 
 From the repository root:
 
-    uv run pytest test/
+    pip install pytest pandas pytest-asyncio pytest-benchmark
+    pytest test/
 
 Run the suite from the root—some tests open fixture files by paths
 relative to it. The full suite processes several real dumps from
-`test/dumps/` and takes a couple of minutes; expected outputs live in
+`test/dumps/`; expected outputs live in `test/baseline_output/`.
-`test/baseline_output/`.
+
+Tests covering `--diff`, `-p wikidiff2`, and `-p legacy` skip when their
+optional dependency is absent, so a base install reports skips rather
+than failures.
 
 ## Authors
 

pyproject.toml

@@ -1,14 +1,34 @@
 [project]
-name = "mediawiki-dump-tools"
+name = "wikiq"
-version = "0.2.0"
+version = "0.3.0"
 description = "Convert MediaWiki XML database dumps into tabular datasets for research"
 readme = "README.md"
 license = "GPL-3.0-or-later"
 license-files = ["COPYING"]
 requires-python = ">=3.11"
+authors = [
+    { name = "Benjamin Mako Hill" },
+    { name = "Nathan TeBlunthuis" },
+    { name = "Will Beason" },
+    { name = "Sohyeon Hwang" },
+    { name = "Kaylea Champion" },
+]
+keywords = ["mediawiki", "wikipedia", "wiki", "dump", "xml", "revision", "research"]
+classifiers = [
+    "Development Status :: 5 - Production/Stable",
+    "Environment :: Console",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)",
+    "Operating System :: POSIX",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Scientific/Engineering :: Information Analysis",
+    "Topic :: Text Processing :: Markup",
+]
 dependencies = [
     "deltas>=0.7.0",
-    "mediawiki-utilities>=0.4.18",
     "more-itertools>=10.7.0",
     "mwparserfromhell>=0.6.0",
     "mwpersistence>=0.2.4",
@@ -16,10 +36,33 @@ dependencies = [
     "mwtypes>=0.4.0",
     "mwxml>=0.3.8",
     "pyarrow>=20.0.0",
-    "pywikidiff2 @ git+https://gitea.communitydata.science/groceryheist/pywikidiff2.git",
     "sortedcontainers>=2.4.0",
 ]
 
+# Two features carry dependencies that are awkward enough to install that they
+# are kept out of the base install: -p legacy needs mediawiki-utilities, a 2015
+# package that also pulls in a MySQL client, and --diff and -p wikidiff2 need
+# pywikidiff2, which compiles a C++ extension. Everything else, including the
+# default -p sequence, works with neither.
+[project.optional-dependencies]
+legacy = ["mediawiki-utilities>=0.4.18"]
+# A wikidiff2 extra belongs here too, but pywikidiff2 is not yet published to
+# PyPI, so an extra naming it could not resolve. Until it is, wikiq reports the
+# git install command when --diff or -p wikidiff2 is used without it.
+
+# Duplicated from [dependency-groups] below, which is PEP 735 and so is visible
+# only to uv. This is what makes `pip install -e '.[dev]'` work.
+dev = [
+    "pandas>=2.1.0",
+    "pytest>=8.4.1",
+    "pytest-asyncio>=1.0.0",
+    "pytest-benchmark>=5.1.0",
+]
+
+[project.urls]
+Homepage = "https://wiki.communitydata.science/"
+Repository = "https://gitea.communitydata.science/collective/wikiq"
+
 [project.scripts]
 wikiq = "wikiq:main"
 
@@ -30,8 +73,18 @@ build-backend = "hatchling.build"
 [tool.hatch.build.targets.wheel]
 packages = ["src/wikiq"]
 
-[tool.hatch.metadata]
+# Without an explicit list hatchling walks the whole working tree, which sweeps
-allow-direct-references = true
+# in any local virtualenv and editor state while still omitting the test dumps,
+# since those are gitignored. Tests run from a clone of the repository rather
+# than from the sdist.
+[tool.hatch.build.targets.sdist]
+include = [
+    "src/wikiq",
+    "README.md",
+    "CHANGELOG.md",
+    "COPYING",
+    "pyproject.toml",
+]
 
 [dependency-groups]
 dev = [

runtest.sh

@@ -1,2 +1,6 @@
 #!/usr/bin/env bash
-uv run pytest test/test_wiki_diff_matcher.py --capture=tee-sys
+# Run the test suite from the repository root, which some tests require
+# because they open fixture files by relative path.
+set -euo pipefail
+cd "$(dirname "$0")"
+exec python -m pytest test/ "$@"

src/wikiq/__init__.py

@@ -32,37 +32,78 @@ from collections import deque, defaultdict
 from hashlib import sha1
 from io import TextIOWrapper
 from itertools import groupby
+from pathlib import Path
 from subprocess import PIPE, Popen
 from typing import IO, Any, Generator, TextIO, Union
 
 import mwpersistence
 import mwreverts
 import mwxml
-import pywikidiff2
+import pyarrow as pa
+import pyarrow.csv as pacsv
+from deltas import SegmentMatcher, SequenceMatcher
 from deltas.tokenizers import wikitext_split
 from more_itertools import peekable
 from mwxml import Dump
+
 import wikiq.tables as tables
-from wikiq.tables import RevisionTable
-from wikiq.wiki_diff_matcher import WikiDiffMatcher
-from wikiq.wikitext_parser import WikitextParser
 from wikiq.resume import get_resume_point
+from wikiq.tables import RevisionTable
+from wikiq.wikitext_parser import WikitextParser
 
 try:
     from importlib.metadata import PackageNotFoundError
     from importlib.metadata import version as _package_version
-    __version__ = _package_version("mediawiki-dump-tools")
+    __version__ = _package_version("wikiq")
 except PackageNotFoundError:
     __version__ = "unknown"
 
 TO_ENCODE = ("title", "editor")
 PERSISTENCE_RADIUS = 7
 DIFF_TIMEOUT_MS = 60000
-from pathlib import Path
 
-import pyarrow as pa
+
-import pyarrow.csv as pacsv
+# Some dependencies serve a single feature and are awkward enough to install
-from deltas import SegmentMatcher, SequenceMatcher
+# that wikiq leaves them out of the base install. Each is imported inside the
+# code path that needs it, so importing wikiq works without them and a plain
+# `pip install` needs no C++ compiler. These helpers do that import and, when
+# it fails, explain what to install rather than raising an ImportError.
+
+def require_pywikidiff2(feature: str):
+    """Import and return pywikidiff2, or exit explaining how to install it.
+
+    feature names the wikiq option that needs it, so the message points at
+    whatever the user actually asked for.
+    """
+    try:
+        import pywikidiff2
+    except ImportError:
+        raise SystemExit(
+            f"{feature} requires pywikidiff2, which wikiq does not install by "
+            "default because it compiles a C++ extension.\n"
+            "Install it with:\n"
+            "    pip install 'pywikidiff2 @ git+"
+            "https://gitea.communitydata.science/groceryheist/pywikidiff2.git'\n"
+            "A C++ compiler and libthai must be available. Other persistence "
+            "methods (-p sequence, -p segment, -p legacy) do not need it."
+        )
+    return pywikidiff2
+
+
+def require_mw_persistence():
+    """Import and return mw.lib.persistence, or exit explaining how to get it."""
+    try:
+        from mw.lib import persistence
+    except ImportError:
+        raise SystemExit(
+            "-p legacy requires mediawiki-utilities, which wikiq no longer "
+            "installs by default.\n"
+            "Install it with:\n"
+            "    pip install 'wikiq[legacy]'\n"
+            "This method exists to reproduce results from older research "
+            "projects; -p sequence is the current equivalent."
+        )
+    return persistence
 
 
 def pyarrow_type_to_spark(pa_type):
@@ -769,6 +810,7 @@ class WikiqParser:
         differ = None
         fast_differ = None
         if self.diff:
+            pywikidiff2 = require_pywikidiff2("--diff")
             differ = pywikidiff2.pywikidiff2(
                 num_context_lines=1000000,
                 max_word_level_diff_complexity=-1,
@@ -856,12 +898,14 @@ class WikiqParser:
                         revert_radius=PERSISTENCE_RADIUS,
                     )
                 elif self.persist == PersistMethod.wikidiff2:
+                    require_pywikidiff2("-p wikidiff2")
+                    from wikiq.wiki_diff_matcher import WikiDiffMatcher
                     wikidiff_matcher = WikiDiffMatcher(tokenizer=wikitext_split)
                     persist_state = mwpersistence.DiffState(
                         wikidiff_matcher, revert_radius=PERSISTENCE_RADIUS
                     )
                 else:
-                    from mw.lib import persistence
+                    persistence = require_mw_persistence()
                     persist_state = persistence.State()
 
             # Pending persistence values waiting for window to fill
@@ -1339,6 +1383,15 @@ def main():
     else:
         persist = PersistMethod.sequence
 
+    # Check for the optional dependencies up front. Both are used deep in the
+    # per-revision loop, and a run can stream for hours before reaching them.
+    if args.diff:
+        require_pywikidiff2("--diff")
+    if persist == PersistMethod.wikidiff2:
+        require_pywikidiff2("-p wikidiff2")
+    elif persist == PersistMethod.legacy:
+        require_mw_persistence()
+
     if args.namespace_filter is not None:
         namespaces = args.namespace_filter
     else:

src/wikiq/wiki_diff_matcher.py

@@ -26,7 +26,6 @@ from mwpersistence import Token
 from sortedcontainers import SortedDict
 
 TOKENIZER = tokenizers.wikitext_split
-import pywikidiff2
 
 
 class DiffToOperationMap:
@@ -348,6 +347,11 @@ class WikiDiffMatcher:
 
     class Processor(DiffEngine.Processor):
         def __init__(self, tokenizer=None):
+            # imported here rather than at module scope so that importing
+            # wikiq does not require the pywikidiff2 C++ extension
+            from wikiq import require_pywikidiff2
+
+            pywikidiff2 = require_pywikidiff2("-p wikidiff2")
             self.tokenizer = tokenizer or TOKENIZER
             self.last_tokens = []
             self.previous_text = ""

test/test_resume.py

@@ -15,6 +15,7 @@ from wikiq_test_utils import (
     TEST_OUTPUT_DIR,
     WIKIQ,
     WikiqTester,
+    requires_pywikidiff2,
 )
 
 
@@ -65,6 +66,7 @@ def test_resume():
     assert_frame_equal(df_full, df_resumed)
 
 
+@requires_pywikidiff2
 def test_resume_with_diff():
     """Test that --resume correctly computes diff values after resume.
 
@@ -305,6 +307,7 @@ def test_resume_data_equivalence():
     assert_frame_equal(df_full, df_resumed)
 
 
+@requires_pywikidiff2
 def test_resume_with_persistence():
     """Test that --resume correctly handles persistence state after resume.
 
@@ -403,6 +406,7 @@ def test_resume_corrupted_jsonl_last_line():
     assert_frame_equal(df_full, df_resumed)
 
 
+@requires_pywikidiff2
 def test_resume_diff_persistence_combined():
     """Test that --resume correctly handles both diff and persistence state together.
 
@@ -448,6 +452,7 @@ def test_resume_diff_persistence_combined():
     assert_frame_equal(df_full, df_resumed)
 
 
+@requires_pywikidiff2
 def test_resume_mid_page():
     """Test resume from the middle of a page with many revisions.
 
@@ -501,6 +506,7 @@ def test_resume_mid_page():
     assert_frame_equal(df_full, df_resumed)
 
 
+@requires_pywikidiff2
 def test_resume_page_boundary():
     """Test resume at the exact start of a new page.
 

test/test_wiki_diff_matcher.py

@@ -1,6 +1,5 @@
 from itertools import chain
 from functools import partial
-import os
 import re
 import pytest
 import pytest_asyncio
@@ -8,6 +7,10 @@ from typing import List
 from deltas import Delete, Equal, Insert, wikitext_split
 from mwpersistence import Token
 from wikiq.wiki_diff_matcher import WikiDiffMatcher
+from wikiq_test_utils import requires_pywikidiff2
+
+# every test here drives wikidiff2 directly
+pytestmark = requires_pywikidiff2
 
 def _replace_whitespace(match):
     if match.group(1):  # If spaces matched (e.g., '  ')
@@ -277,10 +280,6 @@ def test_paragraph_move_and_change():
     assert_equal_enough(a, rev1)
     assert_equal_enough(b, rev2)
 
-@pytest.mark.skipif(
-    not os.path.exists("test/test_diff_revisions/test_infobox_from"),
-    reason="test_infobox_from/_to fixture files are not in the repository",
-)
 def test_infobox():
     rev1 = open("test/test_diff_revisions/test_infobox_from").read()
     rev2 = open("test/test_diff_revisions/test_infobox_to").read()

test/test_wikiq.py

@@ -23,6 +23,8 @@ from wikiq_test_utils import (
     TWINPEAKS,
     WIKIQ,
     WikiqTester,
+    requires_mediawiki_utilities,
+    requires_pywikidiff2,
 )
 
 
@@ -205,6 +207,7 @@ def test_collapse_user():
     baseline = pd.read_table(tester.baseline_file)
     assert_frame_equal(test, baseline, check_like=True)
 
+@requires_pywikidiff2
 def test_pwr_wikidiff2():
     tester = WikiqTester(SAILORMOON, "persistence_wikidiff2", in_compression="7z")
 
@@ -229,6 +232,7 @@ def test_pwr_segment():
     baseline = pd.read_table(tester.baseline_file)
     assert_frame_equal(test, baseline, check_like=True)
 
+@requires_mediawiki_utilities
 def test_pwr_legacy():
     tester = WikiqTester(SAILORMOON, "persistence_legacy", in_compression="7z")
 
@@ -255,6 +259,7 @@ def test_pwr():
     test = test.reindex(columns=sorted(test.columns))
     assert_frame_equal(test, baseline, check_like=True)
 
+@requires_pywikidiff2
 def test_diff():
     tester = WikiqTester(SAILORMOON, "diff", in_compression="7z", out_format='jsonl')
 
@@ -268,6 +273,7 @@ def test_diff():
     assert "diff_timeout" in test.columns, "diff_timeout column should exist"
     assert len(test) > 0, "Should have output rows"
 
+@requires_pywikidiff2
 def test_diff_plus_pwr():
     tester = WikiqTester(SAILORMOON, "diff_pwr", in_compression="7z", out_format='jsonl')
 
@@ -281,6 +287,7 @@ def test_diff_plus_pwr():
     assert "token_revs" in test.columns, "token_revs column should exist"
     assert len(test) > 0, "Should have output rows"
 
+@requires_pywikidiff2
 def test_text():
     tester = WikiqTester(SAILORMOON, "text", in_compression="7z", out_format='jsonl')
 

test/wikiq_test_utils.py

@@ -1,8 +1,27 @@
+import importlib.util
 import os
 import shutil
 import subprocess
 from typing import Final, Union
 
+import pytest
+
+
+def _installed(module: str) -> bool:
+    return importlib.util.find_spec(module) is not None
+
+
+# wikiq's two optional dependencies. Tests exercising --diff, -p wikidiff2, or
+# -p legacy cannot run on a base install, so they skip rather than fail.
+requires_pywikidiff2 = pytest.mark.skipif(
+    not _installed("pywikidiff2"),
+    reason="needs the optional pywikidiff2 extension (--diff, -p wikidiff2)",
+)
+requires_mediawiki_utilities = pytest.mark.skipif(
+    not _installed("mw"),
+    reason="needs the optional mediawiki-utilities package (-p legacy)",
+)
+
 TEST_DIR: Final[str] = os.path.dirname(os.path.realpath(__file__))
 WIKIQ: Final[str] = os.path.join(os.path.join(TEST_DIR, ".."), "src/wikiq/__init__.py")
 TEST_OUTPUT_DIR: Final[str] = os.path.join(TEST_DIR, "test_output")