Benjamin Mako Hill
e4c05caf94
Describes -RPc and -CPc alongside the regex matching options they modify. Kept with the feature rather than on the release branch, so the README there documents only what that branch actually provides. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
128 lines
5.3 KiB
Markdown
128 lines
5.3 KiB
Markdown
# wikiq
|
|
|
|
wikiq converts MediaWiki XML database dumps—the "history" dumps that
|
|
include every revision of every page—into tabular datasets for research.
|
|
It is a command line program that produces one row per revision with
|
|
metadata such as the page, namespace, timestamp, editor, text size, and
|
|
revert status, plus a range of optional computed columns.
|
|
|
|
## Installation
|
|
|
|
wikiq requires Python 3.11 or later. Install it with pip from a clone of
|
|
this repository:
|
|
|
|
git clone https://gitea.communitydata.science/collective/wikiq.git
|
|
cd wikiq
|
|
pip install .
|
|
|
|
Wikimedia dumps are usually compressed as 7z (most common), gz, or bz2.
|
|
wikiq reads these by running your system's decompression tools, so it
|
|
depends on `7za`, `zcat`, and `bzcat` for those respective formats. On
|
|
Debian or Ubuntu, `apt install 7zip` provides `7za`; the others are
|
|
standard.
|
|
|
|
That is everything most uses need, and none of it requires a compiler.
|
|
Two further features carry heavier dependencies, so they are kept out of
|
|
the base install.
|
|
|
|
### Diffs and wikidiff2 persistence
|
|
|
|
`--diff` and `-p wikidiff2` need
|
|
[pywikidiff2](https://gitea.communitydata.science/groceryheist/pywikidiff2),
|
|
a Python binding for MediaWiki's wikidiff2 diff engine. It is not on
|
|
PyPI and compiles a C++ extension, so it needs a C++ compiler and
|
|
libthai:
|
|
|
|
pip install 'pywikidiff2 @ git+https://gitea.communitydata.science/groceryheist/pywikidiff2.git'
|
|
|
|
The other persistence methods, including the default `-p sequence`, do
|
|
not need it.
|
|
|
|
### Legacy persistence
|
|
|
|
`-p legacy` needs `mediawiki-utilities`, which is only useful for
|
|
reproducing results from older research projects:
|
|
|
|
pip install '.[legacy]'
|
|
|
|
wikiq tells you which of these to install if you use an option that
|
|
needs one.
|
|
|
|
## Usage
|
|
|
|
wikiq dump.xml.7z -o output/
|
|
|
|
Each output row describes one revision. The output format follows the
|
|
`-o` argument: a path ending in `.jsonl` produces JSON Lines, anything
|
|
else produces tab-separated values. With no dump file argument, wikiq
|
|
reads XML on stdin and writes to stdout.
|
|
|
|
The `title` and `namespace` columns are page-level identity values as
|
|
of the time of export: a moved page's entire history carries its final
|
|
title, and namespace derives from that title. Redirect status, by
|
|
contrast, is determined per revision from each revision's own text: the
|
|
`revision_is_redirect` and `revision_redirect_target` columns record
|
|
whether a revision's text begins with a redirect directive and where it
|
|
points. `#REDIRECT` is recognized on every wiki; wikis that also use a
|
|
localized keyword can add it with `--redirect-aliases` (for example,
|
|
`--redirect-aliases OMDIRIGERING` for Swedish wikis).
|
|
|
|
The most commonly useful options (`wikiq --help` describes them all):
|
|
|
|
- `-n ID` limits output to a namespace, and can be given more than once.
|
|
`-rr N` sets how many prior edits are checked when detecting reverts.
|
|
- `--collapse-user` collapses each sequence of consecutive edits by the
|
|
same user into a single row, which can address problems with text
|
|
persistence measures.
|
|
- `-p [METHOD]` computes content persistence measures for each revision:
|
|
persistent token revisions, tokens added, and tokens removed. The
|
|
available methods are `sequence` (the default), `segment` (robust to
|
|
content moves but slower), `wikidiff2` (like segment, using the
|
|
wikidiff2 diff engine), and `legacy` (the behavior of older research
|
|
projects). Persistence is the slowest thing wikiq computes.
|
|
- `-d` outputs a structured diff for each revision; `-t` outputs the
|
|
full revision text.
|
|
- `--external-links`, `--citations`, `--wikilinks`, `--templates`, and
|
|
`--headings` parse each revision's wikitext and add a column with the
|
|
extracted elements.
|
|
- `-RP REGEX -RPl LABEL` searches revision text for a regular expression
|
|
and reports the matches in a column named by the label; repeat the
|
|
pair for multiple patterns. `-CP`/`-CPl` do the same for edit
|
|
summaries. Adding `-RPc` or `-CPc` reports the number of matches
|
|
instead of the matched text.
|
|
- `--resume` continues an interrupted run from the last complete line of
|
|
an existing JSONL output file. Combined with `--time-limit HOURS`,
|
|
this supports processing very large dumps in bounded chunks.
|
|
- `--print-schema` prints a Spark-compatible JSON schema for the
|
|
configured output and exits.
|
|
|
|
## Tests
|
|
|
|
From the repository root:
|
|
|
|
pip install pytest pandas pytest-asyncio pytest-benchmark
|
|
pytest test/
|
|
|
|
Run the suite from the root—some tests open fixture files by paths
|
|
relative to it. The full suite processes several real dumps from
|
|
`test/dumps/`; expected outputs live in `test/baseline_output/`.
|
|
|
|
Tests covering `--diff`, `-p wikidiff2`, and `-p legacy` skip when their
|
|
optional dependency is absent, so a base install reports skips rather
|
|
than failures.
|
|
|
|
## Authors
|
|
|
|
wikiq is written and maintained by members of the [Community Data
|
|
Science Collective](https://wiki.communitydata.science/). Contributors
|
|
include Benjamin Mako Hill, Nathan TeBlunthuis, Will Beason, Sohyeon
|
|
Hwang, and Kaylea Champion. It builds on earlier versions of the tool
|
|
written in Python and in C++ by Benjamin Mako Hill.
|
|
|
|
## License
|
|
|
|
wikiq is free software: you can redistribute it and/or modify it under
|
|
the terms of the GNU General Public License as published by the Free
|
|
Software Foundation, either version 3 of the License, or (at your
|
|
option) any later version. See [COPYING](COPYING) for the full text.
|