Files

Benjamin Mako Hill 3d6bf34c8e add --content-sizes: decompose revision content into prose and structured characters

A single mwparserfromhell tree walk per revision emits two nullable
int64 columns. prose_chars counts rendered content outside any template
or tag: text nodes, wikilink labels (or targets when unlabeled),
external link labels, and heading titles. structured_chars counts the
same kinds of content inside template parameter values or tag contents,
with each character assigned by its nearest enclosing container. Markup
syntax, template and tag names, parameter names, attributes, bare URLs,
and HTML comments count in neither, so syntax overhead is derivable as
the total size minus the two columns.

Deleted revisions and parse timeouts yield nulls through the same
machinery as the other parser-based columns and do not change which
revisions are processed. Note that mwparserfromhell parses bold and
italic markup as tags, so emphasized text counts as structured.

Includes hand-computed fixture tests for the counting rules and an
end-to-end test on a dump with deleted revisions verifying nulls and
the invariant prose_chars + structured_chars <= revision size.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

2026-08-06 18:09:29 -07:00

4.2 KiB

Raw Blame History

mediawiki_dump_tools

Tools for converting MediaWiki XML database dumps—the "history" dumps that include every revision of every page—into tabular datasets for research. The main tool is wikiq, a command line program that produces one row per revision with metadata such as the page, namespace, timestamp, editor, text size, and revert status, plus a range of optional computed columns.

Installation

wikiq requires Python 3.11 or later. Install it with pip from a clone of this repository:

git clone https://gitea.communitydata.science/collective/mediawiki_dump_tools.git
cd mediawiki_dump_tools
pip install .

uv works too (uv sync) and is convenient for development, but it is not required.

One dependency, pywikidiff2, is installed from source and compiles a C++ extension, so a C++ compiler must be available at install time.

Wikimedia dumps are usually compressed as 7z (most common), gz, or bz2. wikiq reads these by running your system's decompression tools, so it depends on 7za, zcat, and bzcat for those respective formats. On Debian or Ubuntu, apt install 7zip provides 7za; the others are standard.

Usage

wikiq dump.xml.7z -o output/

Each output row describes one revision. The output format follows the -o argument: a path ending in .jsonl produces JSON Lines, anything else produces tab-separated values. With no dump file argument, wikiq reads XML on stdin and writes to stdout.

The most commonly useful options (wikiq --help describes them all):

-n ID limits output to a namespace, and can be given more than once. -rr N sets how many prior edits are checked when detecting reverts.
--collapse-user collapses each sequence of consecutive edits by the same user into a single row, which can address problems with text persistence measures.
-p [METHOD] computes content persistence measures for each revision: persistent token revisions, tokens added, and tokens removed. The available methods are sequence (the default), segment (robust to content moves but slower), wikidiff2 (like segment, using the wikidiff2 diff engine), and legacy (the behavior of older research projects). Persistence is the slowest thing wikiq computes.
-d outputs a structured diff for each revision; -t outputs the full revision text.
--external-links, --citations, --wikilinks, --templates, and --headings parse each revision's wikitext and add a column with the extracted elements.
--content-sizes counts each revision's rendered content characters in two columns: prose_chars (content outside templates and tags) and structured_chars (template parameter values and tag inner text). Markup syntax and comments count in neither.
-RP REGEX -RPl LABEL searches revision text for a regular expression and reports the matches in a column named by the label; repeat the pair for multiple patterns. -CP/-CPl do the same for edit summaries. Adding -RPc or -CPc reports the number of matches instead of the matched text.
--resume continues an interrupted run from the last complete line of an existing JSONL output file. Combined with --time-limit HOURS, this supports processing very large dumps in bounded chunks.
--print-schema prints a Spark-compatible JSON schema for the configured output and exits.

Tests

From the repository root:

uv run pytest test/

Run the suite from the root—some tests open fixture files by paths relative to it. The full suite processes several real dumps from test/dumps/ and takes around fifteen minutes; expected outputs live in test/baseline_output/.

Authors

wikiq is written and maintained by members of the Community Data Science Collective. Contributors include Benjamin Mako Hill, Nathan TeBlunthuis, Will Beason, Sohyeon Hwang, and Kaylea Champion. It builds on earlier versions of the tool written in Python and in C++ by Benjamin Mako Hill.

License

wikiq is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. See COPYING for the full text.

4.2 KiB Raw Blame History