add --content-sizes: split revisions into content and markup

Wikitravel-style guides get written twice. Someone who does not know
wikitext contributes the information:

    Wonderful Hotel, +15555550123

and someone who does comes along later and polishes it:

    {{listing|name=Wonderful Hotel|phone=+15555550123}}

Both carry the same information. The second adds two dozen characters of
markup around it. --content-sizes measures that split, emitting two
nullable int64 columns from a single mwparserfromhell tree walk per
revision.

content_chars counts what a contributor who did not know wikitext would
have typed: text nodes, template parameter values, tag contents,
wikilink labels (or targets when unlabeled), external link URLs and
their labels, and heading titles. markup_chars is everything else --
braces, pipes, parameter names, tag names and attributes, link targets,
quote marks, and comments, which never reach a reader.

The two columns sum to the revision length, so their ratio reads
directly as how much of a revision is formatting. That invariant is what
the end-to-end test checks on a real dump, along with nulls for deleted
revisions, which come through the same machinery as the other
parser-based columns.

Emphasis needs no special handling: mwparserfromhell represents ''' and
'' as b and i tags, so the quote marks fall out as markup and the words
they wrap as content, which is the wanted answer.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

.gitignore

@@ -4,12 +4,11 @@
 *.xml.xz
 *.swp
 
-# Lockfiles
-uv.lock
-
 # JetBrains
 /.idea
 
+/.claude/settings.local.json*
+
 # Python build and test output
 __pycache__/
 /test/test_output/

.gitmodules

@@ -1,3 +0,0 @@
-[submodule "mediawiki-php-wikidiff2"]
-	path = mediawiki-php-wikidiff2
-	url = https://github.com/wikimedia/mediawiki-php-wikidiff2/

CHANGELOG.md

@@ -0,0 +1,73 @@
+# Changelog
+
+## 0.3.0
+
+Changes relative to what people have been running. Earlier version tags
+in this repository were internal markers rather than releases, so there
+is no previous release to compare against.
+
+### Incompatible changes
+
+- **Redirect detection is now per revision.** The page-level
+  `redirect_target` column has been replaced by `revision_is_redirect`
+  and `revision_redirect_target`, which classify each revision from its
+  own text. The old column reported a page's redirect status as of the
+  time the dump was exported and applied that single value to every
+  revision in the page's history, which is wrong for any page that was
+  turned into a redirect, or turned back, at some point in its life.
+  Revisions with deleted text report null in both columns rather than
+  false. Wikis using a localized redirect keyword can add it with
+  `--redirect-aliases`; `#REDIRECT` is always recognized.
+
+  Code reading `redirect_target` needs updating.
+
+- **Parquet output has been removed**, along with the
+  `--partition-namespaces` and `--max-revisions-per-file` options that
+  only applied to it. Use JSONL, which supports `--resume`.
+
+- **pyspark is no longer a dependency** and the `wikiq-spark` entry point
+  is gone. The second pass of that pipeline was never in this
+  repository.
+
+- **Python 3.11 or later is required.** Python 3.9 reached end of life in
+  October 2025, and the floor unlocks current pyarrow, more-itertools and
+  pandas, along with the CPython 3.11 speedups on the CPU-bound work
+  wikiq does.
+
+- **The project is now called wikiq**, rather than mediawiki_dump_tools.
+
+### Installation
+
+- **wikiq installs with plain `pip` and needs no C++ compiler.**
+  Previously every install required pywikidiff2, built from an ssh-only
+  git URL, so anyone outside the collective could not install the tool at
+  all. It is now optional and needed only for `--diff` and
+  `-p wikidiff2`.
+
+- **`-p legacy` needs the new `legacy` extra.** It was the only user of
+  `mediawiki-utilities`, a 2015 package that also pulls in a MySQL client
+  and six other packages. Everyone was paying for that to support one
+  flag.
+
+- **Dependencies resolve from PyPI** rather than from pinned forks under
+  personal accounts. Requires mwxml >= 0.3.8, which fixes eleven upstream
+  issues; note that mwxml 0.3.7 changed namespace handling to trust the
+  dump's embedded `<ns>` tag.
+
+### Added
+
+- `--version`, and the version is printed to stderr on every run so it
+  lands in job logs beside the output.
+- Licensing: wikiq is GPL-3.0-or-later. See `COPYING`.
+
+### Fixed
+
+- Regex matching (`-RP`/`-CP`) crashed with a `TypeError` on revisions
+  whose text or comment was deleted or suppressed. Deleted content now
+  reports null.
+- Page titles in non-main namespaces were prefixed twice under mwxml
+  0.3.7 and later, producing titles like `Category:Category:Alaska`.
+- The test suite had 16 failing tests, and `pytest test/` silently
+  collected only half of it because the main test file matched neither of
+  pytest's discovery patterns. It is now `test_wikiq.py` and the suite
+  runs green.

README.md

@@ -1,33 +1,53 @@
-# mediawiki_dump_tools
+# wikiq
 
-Tools for converting MediaWiki XML database dumps—the "history" dumps
+wikiq converts MediaWiki XML database dumps—the "history" dumps that
-that include every revision of every page—into tabular datasets for
+include every revision of every page—into tabular datasets for research.
-research. The main tool is `wikiq`, a command line program that produces
+It is a command line program that produces one row per revision with
-one row per revision with metadata such as the page, namespace,
+metadata such as the page, namespace, timestamp, editor, text size, and
-timestamp, editor, text size, and revert status, plus a range of
+revert status, plus a range of optional computed columns.
-optional computed columns.
 
 ## Installation
 
 wikiq requires Python 3.11 or later. Install it with pip from a clone of
 this repository:
 
-    git clone https://gitea.communitydata.science/collective/mediawiki_dump_tools.git
+    git clone https://gitea.communitydata.science/collective/wikiq.git
-    cd mediawiki_dump_tools
+    cd wikiq
     pip install .
 
-[uv](https://docs.astral.sh/uv/) works too (`uv sync`) and is convenient
-for development, but it is not required.
-
-One dependency, `pywikidiff2`, is installed from source and compiles a
-C++ extension, so a C++ compiler must be available at install time.
-
 Wikimedia dumps are usually compressed as 7z (most common), gz, or bz2.
 wikiq reads these by running your system's decompression tools, so it
 depends on `7za`, `zcat`, and `bzcat` for those respective formats. On
 Debian or Ubuntu, `apt install 7zip` provides `7za`; the others are
 standard.
 
+That is everything most uses need, and none of it requires a compiler.
+Two further features carry heavier dependencies, so they are kept out of
+the base install.
+
+### Diffs and wikidiff2 persistence
+
+`--diff` and `-p wikidiff2` need
+[pywikidiff2](https://gitea.communitydata.science/groceryheist/pywikidiff2),
+a Python binding for MediaWiki's wikidiff2 diff engine. It is not on
+PyPI and compiles a C++ extension, so it needs a C++ compiler and
+libthai:
+
+    pip install 'pywikidiff2 @ git+https://gitea.communitydata.science/groceryheist/pywikidiff2.git'
+
+The other persistence methods, including the default `-p sequence`, do
+not need it.
+
+### Legacy persistence
+
+`-p legacy` needs `mediawiki-utilities`, which is only useful for
+reproducing results from older research projects:
+
+    pip install '.[legacy]'
+
+wikiq tells you which of these to install if you use an option that
+needs one.
+
 ## Usage
 
     wikiq dump.xml.7z -o output/
@@ -65,11 +85,17 @@ The most commonly useful options (`wikiq --help` describes them all):
 - `--external-links`, `--citations`, `--wikilinks`, `--templates`, and
   `--headings` parse each revision's wikitext and add a column with the
   extracted elements.
+- `--content-sizes` splits each revision into `content_chars`, the
+  information it carries, and `markup_chars`, the wikitext around it. A
+  contributor who does not know wikitext writes `Wonderful Hotel,
+  +15555550123`; someone polishing it later writes
+  `{{listing|name=Wonderful Hotel|phone=+15555550123}}`. Both carry the
+  same information, and the second adds two dozen characters of markup.
+  The two columns sum to the revision's length.
 - `-RP REGEX -RPl LABEL` searches revision text for a regular expression
   and reports the matches in a column named by the label; repeat the
   pair for multiple patterns. `-CP`/`-CPl` do the same for edit
-  summaries. Adding `-RPc` or `-CPc` reports the number of matches
+  summaries.
-  instead of the matched text.
 - `--resume` continues an interrupted run from the last complete line of
   an existing JSONL output file. Combined with `--time-limit HOURS`,
   this supports processing very large dumps in bounded chunks.
@@ -80,12 +106,16 @@ The most commonly useful options (`wikiq --help` describes them all):
 
 From the repository root:
 
-    uv run pytest test/
+    pip install pytest pandas pytest-asyncio pytest-benchmark
+    pytest test/
 
 Run the suite from the root—some tests open fixture files by paths
 relative to it. The full suite processes several real dumps from
-`test/dumps/` and takes a couple of minutes; expected outputs live in
+`test/dumps/`; expected outputs live in `test/baseline_output/`.
-`test/baseline_output/`.
+
+Tests covering `--diff`, `-p wikidiff2`, and `-p legacy` skip when their
+optional dependency is absent, so a base install reports skips rather
+than failures.
 
 ## Authors
 

pyproject.toml

@@ -1,14 +1,34 @@
 [project]
-name = "mediawiki-dump-tools"
+name = "wikiq"
-version = "0.2.0"
+version = "0.3.0"
 description = "Convert MediaWiki XML database dumps into tabular datasets for research"
 readme = "README.md"
 license = "GPL-3.0-or-later"
 license-files = ["COPYING"]
 requires-python = ">=3.11"
+authors = [
+    { name = "Benjamin Mako Hill" },
+    { name = "Nathan TeBlunthuis" },
+    { name = "Will Beason" },
+    { name = "Sohyeon Hwang" },
+    { name = "Kaylea Champion" },
+]
+keywords = ["mediawiki", "wikipedia", "wiki", "dump", "xml", "revision", "research"]
+classifiers = [
+    "Development Status :: 5 - Production/Stable",
+    "Environment :: Console",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)",
+    "Operating System :: POSIX",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Scientific/Engineering :: Information Analysis",
+    "Topic :: Text Processing :: Markup",
+]
 dependencies = [
     "deltas>=0.7.0",
-    "mediawiki-utilities>=0.4.18",
     "more-itertools>=10.7.0",
     "mwparserfromhell>=0.6.0",
     "mwpersistence>=0.2.4",
@@ -16,10 +36,33 @@ dependencies = [
     "mwtypes>=0.4.0",
     "mwxml>=0.3.8",
     "pyarrow>=20.0.0",
-    "pywikidiff2 @ git+https://gitea.communitydata.science/groceryheist/pywikidiff2.git",
     "sortedcontainers>=2.4.0",
 ]
 
+# Two features carry dependencies that are awkward enough to install that they
+# are kept out of the base install: -p legacy needs mediawiki-utilities, a 2015
+# package that also pulls in a MySQL client, and --diff and -p wikidiff2 need
+# pywikidiff2, which compiles a C++ extension. Everything else, including the
+# default -p sequence, works with neither.
+[project.optional-dependencies]
+legacy = ["mediawiki-utilities>=0.4.18"]
+# A wikidiff2 extra belongs here too, but pywikidiff2 is not yet published to
+# PyPI, so an extra naming it could not resolve. Until it is, wikiq reports the
+# git install command when --diff or -p wikidiff2 is used without it.
+
+# Duplicated from [dependency-groups] below, which is PEP 735 and so is visible
+# only to uv. This is what makes `pip install -e '.[dev]'` work.
+dev = [
+    "pandas>=2.1.0",
+    "pytest>=8.4.1",
+    "pytest-asyncio>=1.0.0",
+    "pytest-benchmark>=5.1.0",
+]
+
+[project.urls]
+Homepage = "https://wiki.communitydata.science/"
+Repository = "https://gitea.communitydata.science/collective/wikiq"
+
 [project.scripts]
 wikiq = "wikiq:main"
 
@@ -30,8 +73,18 @@ build-backend = "hatchling.build"
 [tool.hatch.build.targets.wheel]
 packages = ["src/wikiq"]
 
-[tool.hatch.metadata]
+# Without an explicit list hatchling walks the whole working tree, which sweeps
-allow-direct-references = true
+# in any local virtualenv and editor state while still omitting the test dumps,
+# since those are gitignored. Tests run from a clone of the repository rather
+# than from the sdist.
+[tool.hatch.build.targets.sdist]
+include = [
+    "src/wikiq",
+    "README.md",
+    "CHANGELOG.md",
+    "COPYING",
+    "pyproject.toml",
+]
 
 [dependency-groups]
 dev = [

runtest.sh

@@ -1,2 +1,6 @@
 #!/usr/bin/env bash
-uv run pytest test/test_wiki_diff_matcher.py --capture=tee-sys
+# Run the test suite from the repository root, which some tests require
+# because they open fixture files by relative path.
+set -euo pipefail
+cd "$(dirname "$0")"
+exec python -m pytest test/ "$@"

src/wikiq/__init__.py

@@ -32,37 +32,78 @@ from collections import deque, defaultdict
 from hashlib import sha1
 from io import TextIOWrapper
 from itertools import groupby
+from pathlib import Path
 from subprocess import PIPE, Popen
 from typing import IO, Any, Generator, TextIO, Union
 
 import mwpersistence
 import mwreverts
 import mwxml
-import pywikidiff2
+import pyarrow as pa
+import pyarrow.csv as pacsv
+from deltas import SegmentMatcher, SequenceMatcher
 from deltas.tokenizers import wikitext_split
 from more_itertools import peekable
 from mwxml import Dump
+
 import wikiq.tables as tables
-from wikiq.tables import RevisionTable
-from wikiq.wiki_diff_matcher import WikiDiffMatcher
-from wikiq.wikitext_parser import WikitextParser
 from wikiq.resume import get_resume_point
+from wikiq.tables import RevisionTable
+from wikiq.wikitext_parser import WikitextParser
 
 try:
     from importlib.metadata import PackageNotFoundError
     from importlib.metadata import version as _package_version
-    __version__ = _package_version("mediawiki-dump-tools")
+    __version__ = _package_version("wikiq")
 except PackageNotFoundError:
     __version__ = "unknown"
 
 TO_ENCODE = ("title", "editor")
 PERSISTENCE_RADIUS = 7
 DIFF_TIMEOUT_MS = 60000
-from pathlib import Path
 
-import pyarrow as pa
+
-import pyarrow.csv as pacsv
+# Some dependencies serve a single feature and are awkward enough to install
-from deltas import SegmentMatcher, SequenceMatcher
+# that wikiq leaves them out of the base install. Each is imported inside the
+# code path that needs it, so importing wikiq works without them and a plain
+# `pip install` needs no C++ compiler. These helpers do that import and, when
+# it fails, explain what to install rather than raising an ImportError.
+
+def require_pywikidiff2(feature: str):
+    """Import and return pywikidiff2, or exit explaining how to install it.
+
+    feature names the wikiq option that needs it, so the message points at
+    whatever the user actually asked for.
+    """
+    try:
+        import pywikidiff2
+    except ImportError:
+        raise SystemExit(
+            f"{feature} requires pywikidiff2, which wikiq does not install by "
+            "default because it compiles a C++ extension.\n"
+            "Install it with:\n"
+            "    pip install 'pywikidiff2 @ git+"
+            "https://gitea.communitydata.science/groceryheist/pywikidiff2.git'\n"
+            "A C++ compiler and libthai must be available. Other persistence "
+            "methods (-p sequence, -p segment, -p legacy) do not need it."
+        )
+    return pywikidiff2
+
+
+def require_mw_persistence():
+    """Import and return mw.lib.persistence, or exit explaining how to get it."""
+    try:
+        from mw.lib import persistence
+    except ImportError:
+        raise SystemExit(
+            "-p legacy requires mediawiki-utilities, which wikiq no longer "
+            "installs by default.\n"
+            "Install it with:\n"
+            "    pip install 'wikiq[legacy]'\n"
+            "This method exists to reproduce results from older research "
+            "projects; -p sequence is the current equivalent."
+        )
+    return persistence
 
 
 def pyarrow_type_to_spark(pa_type):
@@ -133,6 +174,7 @@ def build_table(
     wikilinks: bool = False,
     templates: bool = False,
     headings: bool = False,
+    content_sizes: bool = False,
     redirect_aliases: Union[list[str], None] = None,
 ):
     """Build the RevisionTable with appropriate columns based on flags.
@@ -172,7 +214,7 @@ def build_table(
         table.columns.append(tables.RevisionCollapsed())
 
     wikitext_parser = None
-    if external_links or citations or wikilinks or templates or headings:
+    if external_links or citations or wikilinks or templates or headings or content_sizes:
         wikitext_parser = WikitextParser()
 
         if external_links:
@@ -190,6 +232,10 @@ def build_table(
         if headings:
             table.columns.append(tables.RevisionHeadings(wikitext_parser))
 
+        if content_sizes:
+            table.columns.append(tables.RevisionContentChars(wikitext_parser))
+            table.columns.append(tables.RevisionMarkupChars(wikitext_parser))
+
         table.columns.append(tables.RevisionParserTimeout(wikitext_parser))
 
     return table, reverts_column, wikitext_parser
@@ -233,10 +279,10 @@ def build_schema(
     return schema
 
 
-def make_regex_pairs(patterns, labels, count_only=False) -> list:
+def make_regex_pairs(patterns, labels) -> list:
     """Create RegexPair objects from patterns and labels."""
     if (patterns is not None and labels is not None) and (len(patterns) == len(labels)):
-        return [RegexPair(pattern, label, count_only) for pattern, label in zip(patterns, labels)]
+        return [RegexPair(pattern, label) for pattern, label in zip(patterns, labels)]
     elif patterns is None and labels is None:
         return []
     else:
@@ -461,23 +507,21 @@ If the pattern does not include a capture group, then only one output column wil
 
 
 class RegexPair(object):
-    def __init__(self, pattern, label, count_only=False):
+    def __init__(self, pattern, label):
         self.pattern = re.compile(pattern)
         self.label = label
-        self.count_only = count_only
         self.has_groups = bool(self.pattern.groupindex)
         if self.has_groups:
             self.capture_groups = list(self.pattern.groupindex.keys())
 
     def get_pyarrow_fields(self):
-        value_type = pa.int64() if self.count_only else pa.string()
         if self.has_groups:
             fields = [
-                pa.field(self._make_key(cap_group), value_type)
+                pa.field(self._make_key(cap_group), pa.string())
                 for cap_group in self.capture_groups
             ]
         else:
-            fields = [pa.field(self.label, value_type)]
+            fields = [pa.field(self.label, pa.string())]
 
         return fields
 
@@ -490,50 +534,39 @@ class RegexPair(object):
         if self.has_groups:
             # if there are matches of some sort in this revision content, fill the lists for each cap_group
             # content can be None when the text or comment was deleted/suppressed
-            if content is not None and (matchobjects := list(self.pattern.finditer(content))):
+            if content is not None and self.pattern.search(content) is not None:
+                m = self.pattern.finditer(content)
+                matchobjects = list(m)
+
                 for cap_group in self.capture_groups:
                     key = self._make_key(cap_group)
                     temp_list = []
                     for match in matchobjects:
                         # we only want to add the match for the capture group if the match is not None
-                        if (group := match.group(cap_group)) is not None:
+                        if match.group(cap_group) is not None:
-                            temp_list.append(group)
+                            temp_list.append(match.group(cap_group))
 
-                    # if temp_list of matches is empty just make that column None (0 in count mode)
+                    # if temp_list of matches is empty just make that column None
                     if len(temp_list) == 0:
-                        if self.count_only:
+                        temp_dict[key] = None
-                            temp_dict[key] = 0
-                        else:
-                            temp_dict[key] = None
                     # else we put in the list we made in the for-loop above
                     else:
-                        if self.count_only:
+                        temp_dict[key] = ", ".join(temp_list)
-                            temp_dict[key] = len(temp_list)
-                        else:
-                            temp_dict[key] = ", ".join(temp_list)
 
-            # there are no matches at all in this revision content, we default values to None (0 in count mode)
+            # there are no matches at all in this revision content, we default values to None
             else:
                 for cap_group in self.capture_groups:
                     key = self._make_key(cap_group)
-                    if self.count_only:
+                    temp_dict[key] = None
-                        temp_dict[key] = 0
-                    else:
-                        temp_dict[key] = None
 
         # there are no capture groups, we just search for all the matches of the regex
         else:
             # given that there are matches to be made
-            if content is not None and (matches := self.pattern.findall(content)):
+            if content is not None and self.pattern.search(content) is not None:
-                if self.count_only:
+                m = self.pattern.findall(content)
-                    temp_dict[self.label] = len(matches)
+                temp_dict[self.label] = ", ".join(m)
-                else:
-                    temp_dict[self.label] = ", ".join(matches)
             else:
-                if self.count_only:
+                temp_dict[self.label] = None
-                    temp_dict[self.label] = 0
-                else:
-                    temp_dict[self.label] = None
 
         return temp_dict
 
@@ -547,8 +580,6 @@ class WikiqParser:
         regex_match_comment: list[str],
         regex_revision_label: list[str],
         regex_comment_label: list[str],
-        regex_revision_output_count: bool = False,
-        regex_comment_output_count: bool = False,
         text: bool = False,
         diff: bool = False,
         collapse_user: bool = False,
@@ -564,6 +595,7 @@ class WikiqParser:
         wikilinks: bool = False,
         templates: bool = False,
         headings: bool = False,
+        content_sizes: bool = False,
         redirect_aliases: Union[list[str], None] = None,
         time_limit_seconds: Union[float, None] = None,
         input_filename: Union[str, None] = None,
@@ -590,6 +622,7 @@ class WikiqParser:
         self.wikilinks = wikilinks
         self.templates = templates
         self.headings = headings
+        self.content_sizes = content_sizes
         self.redirect_aliases = redirect_aliases
         self.shutdown_requested = False
         self.time_limit_seconds = time_limit_seconds
@@ -600,10 +633,10 @@ class WikiqParser:
 
         self.regex_schemas = []
         self.regex_revision_pairs: list[RegexPair] = self.make_matchmake_pairs(
-            regex_match_revision, regex_revision_label, regex_revision_output_count
+            regex_match_revision, regex_revision_label
         )
         self.regex_comment_pairs: list[RegexPair] = self.make_matchmake_pairs(
-            regex_match_comment, regex_comment_label, regex_comment_output_count
+            regex_match_comment, regex_comment_label
         )
 
         # Initialize output
@@ -646,13 +679,13 @@ class WikiqParser:
         if timer is not None:
             timer.cancel()
 
-    def make_matchmake_pairs(self, patterns, labels, count_only=False) -> list[RegexPair]:
+    def make_matchmake_pairs(self, patterns, labels) -> list[RegexPair]:
         if (patterns is not None and labels is not None) and (
             len(patterns) == len(labels)
         ):
             result: list[RegexPair] = []
             for pattern, label in zip(patterns, labels):
-                rp = RegexPair(pattern, label, count_only)
+                rp = RegexPair(pattern, label)
                 result.append(rp)
                 self.regex_schemas = self.regex_schemas + rp.get_pyarrow_fields()
             return result
@@ -716,6 +749,7 @@ class WikiqParser:
             wikilinks=self.wikilinks,
             templates=self.templates,
             headings=self.headings,
+            content_sizes=self.content_sizes,
             redirect_aliases=self.redirect_aliases,
         )
 
@@ -769,6 +803,7 @@ class WikiqParser:
         differ = None
         fast_differ = None
         if self.diff:
+            pywikidiff2 = require_pywikidiff2("--diff")
             differ = pywikidiff2.pywikidiff2(
                 num_context_lines=1000000,
                 max_word_level_diff_complexity=-1,
@@ -856,12 +891,14 @@ class WikiqParser:
                         revert_radius=PERSISTENCE_RADIUS,
                     )
                 elif self.persist == PersistMethod.wikidiff2:
+                    require_pywikidiff2("-p wikidiff2")
+                    from wikiq.wiki_diff_matcher import WikiDiffMatcher
                     wikidiff_matcher = WikiDiffMatcher(tokenizer=wikitext_split)
                     persist_state = mwpersistence.DiffState(
                         wikidiff_matcher, revert_radius=PERSISTENCE_RADIUS
                     )
                 else:
-                    from mw.lib import persistence
+                    persistence = require_mw_persistence()
                     persist_state = persistence.State()
 
             # Pending persistence values waiting for window to fill
@@ -1191,14 +1228,6 @@ def main():
         help="The label for the outputted column based on matching the regex in revision text.",
     )
 
-    parser.add_argument(
-        "-RPc",
-        "--revision-pattern-count",
-        dest="regex_revision_output_count",
-        action="store_true",
-        help="If present, this will cause the revision patterns to return counts of the number of matches instead of the text of the matches themselves. It will affect all revision patterns.",
-    )
-
     parser.add_argument(
         "-CP",
         "--comment-pattern",
@@ -1219,14 +1248,6 @@ def main():
         help="The label for the outputted column based on matching the regex in comments.",
     )
 
-    parser.add_argument(
-        "-CPc",
-        "--comment-pattern-count",
-        dest="regex_comment_output_count",
-        action="store_true",
-        help="If present, this will cause the comment patterns to return counts of the number of matches instead of the text of the matches themselves. It will affect all comment patterns.",
-    )
-
     parser.add_argument(
         "-d",
         "--diff",
@@ -1285,6 +1306,14 @@ def main():
         help="Extract section headings from each revision.",
     )
 
+    parser.add_argument(
+        "--content-sizes",
+        dest="content_sizes",
+        action="store_true",
+        default=False,
+        help="Split each revision into content_chars, the information it carries, and markup_chars, the wikitext formatting around it. Content is what someone who did not know wikitext would have typed: words, link labels, and template parameter values. The two columns sum to the revision length.",
+    )
+
     parser.add_argument(
         "--redirect-aliases",
         dest="redirect_aliases",
@@ -1339,6 +1368,15 @@ def main():
     else:
         persist = PersistMethod.sequence
 
+    # Check for the optional dependencies up front. Both are used deep in the
+    # per-revision loop, and a run can stream for hours before reaching them.
+    if args.diff:
+        require_pywikidiff2("--diff")
+    if persist == PersistMethod.wikidiff2:
+        require_pywikidiff2("-p wikidiff2")
+    elif persist == PersistMethod.legacy:
+        require_mw_persistence()
+
     if args.namespace_filter is not None:
         namespaces = args.namespace_filter
     else:
@@ -1356,8 +1394,8 @@ def main():
 
     # Handle --print-schema: build and output schema, then exit
     if args.print_schema:
-        regex_revision_pairs = make_regex_pairs(args.regex_match_revision, args.regex_revision_label, args.regex_revision_output_count)
+        regex_revision_pairs = make_regex_pairs(args.regex_match_revision, args.regex_revision_label)
-        regex_comment_pairs = make_regex_pairs(args.regex_match_comment, args.regex_comment_label, args.regex_comment_output_count)
+        regex_comment_pairs = make_regex_pairs(args.regex_match_comment, args.regex_comment_label)
 
         table, _, _ = build_table(
             text=args.text,
@@ -1367,6 +1405,7 @@ def main():
             wikilinks=args.wikilinks,
             templates=args.templates,
             headings=args.headings,
+            content_sizes=args.content_sizes,
             redirect_aliases=redirect_aliases,
         )
         schema = build_schema(
@@ -1435,10 +1474,8 @@ def main():
                 revert_radius=args.revert_radius,
                 regex_match_revision=args.regex_match_revision,
                 regex_revision_label=args.regex_revision_label,
-                regex_revision_output_count=args.regex_revision_output_count,
                 regex_match_comment=args.regex_match_comment,
                 regex_comment_label=args.regex_comment_label,
-                regex_comment_output_count=args.regex_comment_output_count,
                 text=args.text,
                 diff=args.diff,
                 output_jsonl=output_jsonl,
@@ -1450,6 +1487,7 @@ def main():
                 wikilinks=args.wikilinks,
                 templates=args.templates,
                 headings=args.headings,
+                content_sizes=args.content_sizes,
                 redirect_aliases=redirect_aliases,
                 time_limit_seconds=time_limit_seconds,
                 input_filename=filename,
@@ -1494,10 +1532,8 @@ def main():
             revert_radius=args.revert_radius,
             regex_match_revision=args.regex_match_revision,
             regex_revision_label=args.regex_revision_label,
-            regex_revision_output_count=args.regex_revision_output_count,
             regex_match_comment=args.regex_match_comment,
             regex_comment_label=args.regex_comment_label,
-            regex_comment_output_count=args.regex_comment_output_count,
             diff=args.diff,
             text=args.text,
             batch_size=args.batch_size,
@@ -1507,6 +1543,7 @@ def main():
             wikilinks=args.wikilinks,
             templates=args.templates,
             headings=args.headings,
+            content_sizes=args.content_sizes,
             redirect_aliases=redirect_aliases,
             time_limit_seconds=time_limit_seconds,
         )

src/wikiq/tables.py

@@ -384,6 +384,50 @@ class RevisionHeadings(RevisionField[Union[list[dict], None]]):
         return self.wikitext_parser.extract_headings(revision.text)
 
 
+class RevisionContentChars(RevisionField[Union[int, None]]):
+    """Count the characters of information a revision carries.
+
+    The words, link labels and template parameter values -- what someone
+    who did not know wikitext would have typed. See
+    WikitextParser.content_sizes().
+    """
+
+    field = pa.field("content_chars", pa.int64(), nullable=True)
+
+    def __init__(self, wikitext_parser: "WikitextParser"):
+        super().__init__()
+        self.wikitext_parser = wikitext_parser
+
+    def extract(self, page: mwtypes.Page, revisions: list[mwxml.Revision]) -> Union[int, None]:
+        revision = revisions[-1]
+        if revision.deleted.text:
+            return None
+        sizes = self.wikitext_parser.content_sizes(revision.text)
+        return None if sizes is None else sizes[0]
+
+
+class RevisionMarkupChars(RevisionField[Union[int, None]]):
+    """Count the characters of markup wrapped around a revision's content.
+
+    Braces, pipes, parameter names, tag names and attributes, link targets
+    and comments. Together with content_chars this sums to the revision
+    length.
+    """
+
+    field = pa.field("markup_chars", pa.int64(), nullable=True)
+
+    def __init__(self, wikitext_parser: "WikitextParser"):
+        super().__init__()
+        self.wikitext_parser = wikitext_parser
+
+    def extract(self, page: mwtypes.Page, revisions: list[mwxml.Revision]) -> Union[int, None]:
+        revision = revisions[-1]
+        if revision.deleted.text:
+            return None
+        sizes = self.wikitext_parser.content_sizes(revision.text)
+        return None if sizes is None else sizes[1]
+
+
 class RevisionParserTimeout(RevisionField[bool]):
     """Track whether the wikitext parser timed out for this revision."""
 

src/wikiq/wiki_diff_matcher.py

@@ -26,7 +26,6 @@ from mwpersistence import Token
 from sortedcontainers import SortedDict
 
 TOKENIZER = tokenizers.wikitext_split
-import pywikidiff2
 
 
 class DiffToOperationMap:
@@ -348,6 +347,11 @@ class WikiDiffMatcher:
 
     class Processor(DiffEngine.Processor):
         def __init__(self, tokenizer=None):
+            # imported here rather than at module scope so that importing
+            # wikiq does not require the pywikidiff2 C++ extension
+            from wikiq import require_pywikidiff2
+
+            pywikidiff2 = require_pywikidiff2("-p wikidiff2")
             self.tokenizer = tokenizer or TOKENIZER
             self.last_tokens = []
             self.previous_text = ""

src/wikiq/wikitext_parser.py

@@ -20,6 +20,17 @@ from __future__ import annotations
 import signal
 
 import mwparserfromhell
+from mwparserfromhell.nodes import (
+    Argument,
+    Comment,
+    ExternalLink,
+    Heading,
+    HTMLEntity,
+    Tag,
+    Template,
+    Text,
+    Wikilink,
+)
 
 PARSER_TIMEOUT = 60  # seconds
 
@@ -38,6 +49,8 @@ class WikitextParser:
         self._cached_text: str | None = None
         self._cached_wikicode = None
         self.last_parse_timed_out: bool = False
+        self._sizes_text: str | None = None
+        self._cached_sizes: tuple[int, int] | None = None
 
     def _timeout_handler(self, signum, frame):
         raise TimeoutError("mwparserfromhell parse exceeded timeout")
@@ -146,6 +159,85 @@ class WikitextParser:
         except Exception:
             return None
 
+    def _count_content(self, wikicode, counts: list[int]) -> None:
+        """Walk a wikicode tree adding the length of rendered content.
+
+        counts is a one-element list holding the running content total.
+        Anything not counted here is markup; content_sizes() derives the
+        markup total by subtraction, so every character lands in one column
+        or the other.
+        """
+        for node in wikicode.nodes:
+            if isinstance(node, Text):
+                counts[0] += len(node.value)
+            elif isinstance(node, HTMLEntity):
+                # an entity stands in for the character it renders as, so
+                # only that character is content and the escaping is markup
+                counts[0] += len(node.normalize())
+            elif isinstance(node, Comment):
+                # never reaches a reader, so it is not information
+                continue
+            elif isinstance(node, Template):
+                # the parameter values are the information; the template
+                # name, parameter names and delimiters are the polish that
+                # someone later wrapped around it
+                for param in node.params:
+                    self._count_content(param.value, counts)
+            elif isinstance(node, Tag):
+                # tag contents are information; the tag name and attributes
+                # are markup. This covers ''italic'' and '''bold''', which
+                # mwparserfromhell represents as i and b tags
+                if node.contents is not None:
+                    self._count_content(node.contents, counts)
+            elif isinstance(node, Wikilink):
+                # the label if present, otherwise the target, is what a
+                # reader sees; the brackets and the link target are markup
+                rendered = node.text if node.text is not None else node.title
+                self._count_content(rendered, counts)
+            elif isinstance(node, ExternalLink):
+                # both halves are information someone typed: a contributor
+                # who did not know wikitext would still have written the URL
+                counts[0] += len(str(node.url))
+                if node.title is not None:
+                    self._count_content(node.title, counts)
+            elif isinstance(node, Heading):
+                self._count_content(node.title, counts)
+            elif isinstance(node, Argument):
+                if node.default is not None:
+                    self._count_content(node.default, counts)
+
+    def content_sizes(self, text: str | None) -> tuple[int, int] | None:
+        """Split a revision into the information it carries and the markup
+        around it.
+
+        Returns (content_chars, markup_chars), which sum to len(text).
+
+        Content is what a contributor who did not know wikitext would have
+        typed: the words, the link labels, the values inside template
+        parameters. Markup is everything a later editor added to format it —
+        braces, pipes, parameter names, tag names and attributes, link
+        targets, quote marks, and comments.
+
+        So "Wonderful Hotel, +15555550123" is almost all content, while
+        "{{listing|name=Wonderful Hotel|phone=+15555550123}}" carries the
+        same information with two dozen characters of markup around it.
+        """
+        if text is None:
+            return None
+        if text == self._sizes_text:
+            return self._cached_sizes
+        try:
+            wikicode = self._get_wikicode(text)
+            counts = [0]
+            self._count_content(wikicode, counts)
+            content = min(counts[0], len(text))
+            result = (content, len(text) - content)
+        except Exception:
+            result = None
+        self._sizes_text = text
+        self._cached_sizes = result
+        return result
+
     def extract_headings(self, text: str | None) -> list[dict] | None:
         """Extract all section headings with their levels."""
         if text is None:

test/test_resume.py

@@ -15,6 +15,7 @@ from wikiq_test_utils import (
     TEST_OUTPUT_DIR,
     WIKIQ,
     WikiqTester,
+    requires_pywikidiff2,
 )
 
 
@@ -65,6 +66,7 @@ def test_resume():
     assert_frame_equal(df_full, df_resumed)
 
 
+@requires_pywikidiff2
 def test_resume_with_diff():
     """Test that --resume correctly computes diff values after resume.
 
@@ -305,6 +307,7 @@ def test_resume_data_equivalence():
     assert_frame_equal(df_full, df_resumed)
 
 
+@requires_pywikidiff2
 def test_resume_with_persistence():
     """Test that --resume correctly handles persistence state after resume.
 
@@ -403,6 +406,7 @@ def test_resume_corrupted_jsonl_last_line():
     assert_frame_equal(df_full, df_resumed)
 
 
+@requires_pywikidiff2
 def test_resume_diff_persistence_combined():
     """Test that --resume correctly handles both diff and persistence state together.
 
@@ -448,6 +452,7 @@ def test_resume_diff_persistence_combined():
     assert_frame_equal(df_full, df_resumed)
 
 
+@requires_pywikidiff2
 def test_resume_mid_page():
     """Test resume from the middle of a page with many revisions.
 
@@ -501,6 +506,7 @@ def test_resume_mid_page():
     assert_frame_equal(df_full, df_resumed)
 
 
+@requires_pywikidiff2
 def test_resume_page_boundary():
     """Test resume at the exact start of a new page.
 

test/test_wiki_diff_matcher.py

@@ -1,6 +1,5 @@
 from itertools import chain
 from functools import partial
-import os
 import re
 import pytest
 import pytest_asyncio
@@ -8,6 +7,10 @@ from typing import List
 from deltas import Delete, Equal, Insert, wikitext_split
 from mwpersistence import Token
 from wikiq.wiki_diff_matcher import WikiDiffMatcher
+from wikiq_test_utils import requires_pywikidiff2
+
+# every test here drives wikidiff2 directly
+pytestmark = requires_pywikidiff2
 
 def _replace_whitespace(match):
     if match.group(1):  # If spaces matched (e.g., '  ')
@@ -277,10 +280,6 @@ def test_paragraph_move_and_change():
     assert_equal_enough(a, rev1)
     assert_equal_enough(b, rev2)
 
-@pytest.mark.skipif(
-    not os.path.exists("test/test_diff_revisions/test_infobox_from"),
-    reason="test_infobox_from/_to fixture files are not in the repository",
-)
 def test_infobox():
     rev1 = open("test/test_diff_revisions/test_infobox_from").read()
     rev2 = open("test/test_diff_revisions/test_infobox_to").read()

test/test_wikiq.py

@@ -12,7 +12,7 @@ import pytest
 from pandas import DataFrame
 from pandas.testing import assert_frame_equal, assert_series_equal
 
-from wikiq import build_table, build_schema, RegexPair
+from wikiq import build_table, build_schema, RegexPair, WikitextParser
 from wikiq_test_utils import (
     BASELINE_DIR,
     IKWIKI,
@@ -23,6 +23,8 @@ from wikiq_test_utils import (
     TWINPEAKS,
     WIKIQ,
     WikiqTester,
+    requires_mediawiki_utilities,
+    requires_pywikidiff2,
 )
 
 
@@ -205,6 +207,7 @@ def test_collapse_user():
     baseline = pd.read_table(tester.baseline_file)
     assert_frame_equal(test, baseline, check_like=True)
 
+@requires_pywikidiff2
 def test_pwr_wikidiff2():
     tester = WikiqTester(SAILORMOON, "persistence_wikidiff2", in_compression="7z")
 
@@ -229,6 +232,7 @@ def test_pwr_segment():
     baseline = pd.read_table(tester.baseline_file)
     assert_frame_equal(test, baseline, check_like=True)
 
+@requires_mediawiki_utilities
 def test_pwr_legacy():
     tester = WikiqTester(SAILORMOON, "persistence_legacy", in_compression="7z")
 
@@ -255,6 +259,7 @@ def test_pwr():
     test = test.reindex(columns=sorted(test.columns))
     assert_frame_equal(test, baseline, check_like=True)
 
+@requires_pywikidiff2
 def test_diff():
     tester = WikiqTester(SAILORMOON, "diff", in_compression="7z", out_format='jsonl')
 
@@ -268,6 +273,7 @@ def test_diff():
     assert "diff_timeout" in test.columns, "diff_timeout column should exist"
     assert len(test) > 0, "Should have output rows"
 
+@requires_pywikidiff2
 def test_diff_plus_pwr():
     tester = WikiqTester(SAILORMOON, "diff_pwr", in_compression="7z", out_format='jsonl')
 
@@ -281,6 +287,7 @@ def test_diff_plus_pwr():
     assert "token_revs" in test.columns, "token_revs column should exist"
     assert len(test) > 0, "Should have output rows"
 
+@requires_pywikidiff2
 def test_text():
     tester = WikiqTester(SAILORMOON, "text", in_compression="7z", out_format='jsonl')
 
@@ -420,55 +427,6 @@ def test_regex_deleted_revisions():
     assert deleted["npov_npov"].isna().all()
     assert deleted["talk_talk"].isna().all()
 
-def test_regex_count_only():
-    # count mode returns match counts instead of the matched text, with 0
-    # (never None) for revisions with no matches or deleted content
-    pair = RegexPair(r"\bcat\b", "cats", count_only=True)
-    assert pair.matchmake("cat cat dog cat") == {"cats": 3}
-    assert pair.matchmake("dog") == {"cats": 0}
-    assert pair.matchmake(None) == {"cats": 0}
-    assert pair.get_pyarrow_fields()[0].type == pa.int64()
-
-    pair = RegexPair(r"(?P<a>\bcat\b)|(?P<b>\bdog\b)", "pets", count_only=True)
-    assert pair.matchmake("cat dog cat") == {"pets_a": 2, "pets_b": 1}
-    assert pair.matchmake("bird") == {"pets_a": 0, "pets_b": 0}
-    assert pair.matchmake(None) == {"pets_a": 0, "pets_b": 0}
-    assert all(f.type == pa.int64() for f in pair.get_pyarrow_fields())
-
-    # without count_only the original behavior is unchanged
-    pair = RegexPair(r"\bcat\b", "cats")
-    assert pair.matchmake("cat cat dog cat") == {"cats": "cat, cat, cat"}
-    assert pair.get_pyarrow_fields()[0].type == pa.string()
-
-def test_regex_count_e2e():
-    # -RPc/-CPc produce integer columns whose values equal the number of
-    # matches in the revision text and comment respectively
-    tester = WikiqTester(wiki=REGEXTEST, case_name="regex_count")
-
-    try:
-        tester.call_wikiq(
-            "--text",
-            "-RP 'TestCase' -RPl testcases -RPc",
-            "-CP '\\b[Cc]hevalier\\b' -CPl chev -CPc",
-        )
-    except subprocess.CalledProcessError as exc:
-        pytest.fail(exc.stderr.decode("utf8"))
-
-    import re
-    test = pd.read_table(tester.output)
-
-    assert test["testcases"].dtype == "int64"
-    assert test["chev"].dtype == "int64"
-
-    for _, row in test.iterrows():
-        text = row["text"] if isinstance(row["text"], str) else None
-        expected = len(re.findall("TestCase", text)) if text is not None else 0
-        assert row["testcases"] == expected, f"revid {row['revid']}: {row['testcases']} != {expected}"
-
-        comment = row["edit_summary"] if isinstance(row["edit_summary"], str) else None
-        expected = len(re.findall(r"\b[Cc]hevalier\b", comment)) if comment is not None else 0
-        assert row["chev"] == expected, f"revid {row['revid']}: {row['chev']} != {expected}"
-
 def test_redirect_detection():
     from wikiq.tables import RedirectDetector
 
@@ -526,6 +484,71 @@ def test_redirect_columns_e2e():
     assert len(redirects) > 0
     assert redirects["revision_redirect_target"].notna().all()
 
+def test_content_sizes():
+    # hand-computed fixtures: (text, content_chars, markup_chars)
+    cases = [
+        # plain text is all content
+        ("Hello world.", 12, 0),
+        # the same information written by someone who knows wikitext: the
+        # values survive as content, the scaffolding around them is markup
+        ("Wonderful Hotel, +15555550123", 29, 0),
+        ("{{listing|name=Wonderful Hotel|phone=+15555550123}}", 27, 24),
+        # emphasis is markup applied to content, not content itself
+        ("''whatever''", 8, 4),
+        ("Some '''bold''' prose.", 16, 6),
+        # tag contents are content; the tag name and attributes are not
+        ('Before <span class="x">inner text</span> after', 23, 23),
+        # a wikilink renders its label, so the target is markup
+        ("[[Main Page|the main page]]", 13, 14),
+        # an unlabeled wikilink renders its target, which is then content
+        ("[[Help]]", 4, 4),
+        # both halves of an external link are information someone typed
+        ("[http://x.com label]", 17, 3),
+        # comments never reach a reader, so they are entirely markup
+        ("Text <!-- hidden --> more", 10, 15),
+        # nesting: values at any depth are content
+        ("{{outer|a={{inner|b=deep}} shallow}}", 12, 24),
+    ]
+
+    parser = WikitextParser()
+    for text, content, markup in cases:
+        result = parser.content_sizes(text)
+        assert result == (content, markup), f"{text!r}: got {result}, want {(content, markup)}"
+        # every character lands in exactly one column
+        assert result[0] + result[1] == len(text), f"{text!r}: columns do not sum to length"
+
+    assert parser.content_sizes(None) is None
+
+def test_content_sizes_e2e():
+    # the ikwiki dump contains revisions with deleted text, which must get
+    # null content sizes; for every other revision the two columns must
+    # account for the whole revision
+    tester = WikiqTester(IKWIKI, "content_sizes")
+
+    try:
+        tester.call_wikiq("--content-sizes", "--text")
+    except subprocess.CalledProcessError as exc:
+        pytest.fail(exc.stderr.decode("utf8"))
+
+    test = pd.read_table(tester.output)
+
+    assert "content_chars" in test.columns
+    assert "markup_chars" in test.columns
+
+    deleted = test[test["deleted"]]
+    assert len(deleted) > 0
+    assert deleted["content_chars"].isna().all()
+    assert deleted["markup_chars"].isna().all()
+
+    present = test[~test["deleted"] & test["text"].notna() & test["content_chars"].notna()]
+    assert len(present) > 0
+    total = present["content_chars"] + present["markup_chars"]
+    assert (total == present["text"].str.len()).all()
+    # a real wiki has both kinds of revision: some almost pure text, some
+    # carrying a lot of markup
+    assert (present["markup_chars"] == 0).any()
+    assert (present["markup_chars"] > 0).any()
+
 def test_external_links_only():
     """Test that --external-links extracts external links correctly."""
     import mwparserfromhell

test/wikiq_test_utils.py

@@ -1,8 +1,27 @@
+import importlib.util
 import os
 import shutil
 import subprocess
 from typing import Final, Union
 
+import pytest
+
+
+def _installed(module: str) -> bool:
+    return importlib.util.find_spec(module) is not None
+
+
+# wikiq's two optional dependencies. Tests exercising --diff, -p wikidiff2, or
+# -p legacy cannot run on a base install, so they skip rather than fail.
+requires_pywikidiff2 = pytest.mark.skipif(
+    not _installed("pywikidiff2"),
+    reason="needs the optional pywikidiff2 extension (--diff, -p wikidiff2)",
+)
+requires_mediawiki_utilities = pytest.mark.skipif(
+    not _installed("mw"),
+    reason="needs the optional mediawiki-utilities package (-p legacy)",
+)
+
 TEST_DIR: Final[str] = os.path.dirname(os.path.realpath(__file__))
 WIKIQ: Final[str] = os.path.join(os.path.join(TEST_DIR, ".."), "src/wikiq/__init__.py")
 TEST_OUTPUT_DIR: Final[str] = os.path.join(TEST_DIR, "test_output")