Benjamin Mako Hill
0fa8b3d9ba
The installation section led with the two optional dependencies and left the decompression tools until last, which put the awkward parts first and buried something every run actually needs. Describe the base install and the compression handling together, then the optional features under their own headings. Drop the note about uv. It said only that uv also works, which anyone who uses uv already knows. Also open with what wikiq is rather than describing the repository as a collection of tools, now that the repository is named after the program. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
127 lines
5.2 KiB
Markdown
127 lines
5.2 KiB
Markdown
# wikiq
|
|
|
|
wikiq converts MediaWiki XML database dumps—the "history" dumps that
|
|
include every revision of every page—into tabular datasets for research.
|
|
It is a command line program that produces one row per revision with
|
|
metadata such as the page, namespace, timestamp, editor, text size, and
|
|
revert status, plus a range of optional computed columns.
|
|
|
|
## Installation
|
|
|
|
wikiq requires Python 3.11 or later. Install it with pip from a clone of
|
|
this repository:
|
|
|
|
git clone https://gitea.communitydata.science/collective/wikiq.git
|
|
cd wikiq
|
|
pip install .
|
|
|
|
Wikimedia dumps are usually compressed as 7z (most common), gz, or bz2.
|
|
wikiq reads these by running your system's decompression tools, so it
|
|
depends on `7za`, `zcat`, and `bzcat` for those respective formats. On
|
|
Debian or Ubuntu, `apt install 7zip` provides `7za`; the others are
|
|
standard.
|
|
|
|
That is everything most uses need, and none of it requires a compiler.
|
|
Two further features carry heavier dependencies, so they are kept out of
|
|
the base install.
|
|
|
|
### Diffs and wikidiff2 persistence
|
|
|
|
`--diff` and `-p wikidiff2` need
|
|
[pywikidiff2](https://gitea.communitydata.science/groceryheist/pywikidiff2),
|
|
a Python binding for MediaWiki's wikidiff2 diff engine. It is not on
|
|
PyPI and compiles a C++ extension, so it needs a C++ compiler and
|
|
libthai:
|
|
|
|
pip install 'pywikidiff2 @ git+https://gitea.communitydata.science/groceryheist/pywikidiff2.git'
|
|
|
|
The other persistence methods, including the default `-p sequence`, do
|
|
not need it.
|
|
|
|
### Legacy persistence
|
|
|
|
`-p legacy` needs `mediawiki-utilities`, which is only useful for
|
|
reproducing results from older research projects:
|
|
|
|
pip install '.[legacy]'
|
|
|
|
wikiq tells you which of these to install if you use an option that
|
|
needs one.
|
|
|
|
## Usage
|
|
|
|
wikiq dump.xml.7z -o output/
|
|
|
|
Each output row describes one revision. The output format follows the
|
|
`-o` argument: a path ending in `.jsonl` produces JSON Lines, anything
|
|
else produces tab-separated values. With no dump file argument, wikiq
|
|
reads XML on stdin and writes to stdout.
|
|
|
|
The `title` and `namespace` columns are page-level identity values as
|
|
of the time of export: a moved page's entire history carries its final
|
|
title, and namespace derives from that title. Redirect status, by
|
|
contrast, is determined per revision from each revision's own text: the
|
|
`revision_is_redirect` and `revision_redirect_target` columns record
|
|
whether a revision's text begins with a redirect directive and where it
|
|
points. `#REDIRECT` is recognized on every wiki; wikis that also use a
|
|
localized keyword can add it with `--redirect-aliases` (for example,
|
|
`--redirect-aliases OMDIRIGERING` for Swedish wikis).
|
|
|
|
The most commonly useful options (`wikiq --help` describes them all):
|
|
|
|
- `-n ID` limits output to a namespace, and can be given more than once.
|
|
`-rr N` sets how many prior edits are checked when detecting reverts.
|
|
- `--collapse-user` collapses each sequence of consecutive edits by the
|
|
same user into a single row, which can address problems with text
|
|
persistence measures.
|
|
- `-p [METHOD]` computes content persistence measures for each revision:
|
|
persistent token revisions, tokens added, and tokens removed. The
|
|
available methods are `sequence` (the default), `segment` (robust to
|
|
content moves but slower), `wikidiff2` (like segment, using the
|
|
wikidiff2 diff engine), and `legacy` (the behavior of older research
|
|
projects). Persistence is the slowest thing wikiq computes.
|
|
- `-d` outputs a structured diff for each revision; `-t` outputs the
|
|
full revision text.
|
|
- `--external-links`, `--citations`, `--wikilinks`, `--templates`, and
|
|
`--headings` parse each revision's wikitext and add a column with the
|
|
extracted elements.
|
|
- `-RP REGEX -RPl LABEL` searches revision text for a regular expression
|
|
and reports the matches in a column named by the label; repeat the
|
|
pair for multiple patterns. `-CP`/`-CPl` do the same for edit
|
|
summaries.
|
|
- `--resume` continues an interrupted run from the last complete line of
|
|
an existing JSONL output file. Combined with `--time-limit HOURS`,
|
|
this supports processing very large dumps in bounded chunks.
|
|
- `--print-schema` prints a Spark-compatible JSON schema for the
|
|
configured output and exits.
|
|
|
|
## Tests
|
|
|
|
From the repository root:
|
|
|
|
pip install pytest pandas pytest-asyncio pytest-benchmark
|
|
pytest test/
|
|
|
|
Run the suite from the root—some tests open fixture files by paths
|
|
relative to it. The full suite processes several real dumps from
|
|
`test/dumps/`; expected outputs live in `test/baseline_output/`.
|
|
|
|
Tests covering `--diff`, `-p wikidiff2`, and `-p legacy` skip when their
|
|
optional dependency is absent, so a base install reports skips rather
|
|
than failures.
|
|
|
|
## Authors
|
|
|
|
wikiq is written and maintained by members of the [Community Data
|
|
Science Collective](https://wiki.communitydata.science/). Contributors
|
|
include Benjamin Mako Hill, Nathan TeBlunthuis, Will Beason, Sohyeon
|
|
Hwang, and Kaylea Champion. It builds on earlier versions of the tool
|
|
written in Python and in C++ by Benjamin Mako Hill.
|
|
|
|
## License
|
|
|
|
wikiq is free software: you can redistribute it and/or modify it under
|
|
the terms of the GNU General Public License as published by the Free
|
|
Software Foundation, either version 3 of the License, or (at your
|
|
option) any later version. See [COPYING](COPYING) for the full text.
|