wikiq/README.md

# wikiq

wikiq converts MediaWiki XML database dumps—the "history" dumps that
include every revision of every page—into tabular datasets for research.
It is a command line program that produces one row per revision with
metadata such as the page, namespace, timestamp, editor, text size, and
revert status, plus a range of optional computed columns.

## Installation

wikiq requires Python 3.11 or later. Install it with pip from a clone of
this repository:

    git clone https://gitea.communitydata.science/collective/wikiq.git
    cd wikiq
    pip install .

Wikimedia dumps are usually compressed as 7z (most common), gz, or bz2.
wikiq reads these by running your system's decompression tools, so it
depends on `7za`, `zcat`, and `bzcat` for those respective formats. On
Debian or Ubuntu, `apt install 7zip` provides `7za`; the others are
standard.

That is everything most uses need, and none of it requires a compiler.
Two further features carry heavier dependencies, so they are kept out of
the base install.

### Diffs and wikidiff2 persistence

`--diff` and `-p wikidiff2` need
[pywikidiff2](https://gitea.communitydata.science/groceryheist/pywikidiff2),
a Python binding for MediaWiki's wikidiff2 diff engine. It is not on
PyPI and compiles a C++ extension, so it needs a C++ compiler and
libthai:

    pip install 'pywikidiff2 @ git+https://gitea.communitydata.science/groceryheist/pywikidiff2.git'

The other persistence methods, including the default `-p sequence`, do
not need it.

### Legacy persistence

`-p legacy` needs `mediawiki-utilities`, which is only useful for
reproducing results from older research projects:

    pip install '.[legacy]'

wikiq tells you which of these to install if you use an option that
needs one.

## Usage

    wikiq dump.xml.7z -o output/

Each output row describes one revision. The output format follows the
`-o` argument: a path ending in `.jsonl` produces JSON Lines, anything
else produces tab-separated values. With no dump file argument, wikiq
reads XML on stdin and writes to stdout.

The `title` and `namespace` columns are page-level identity values as
of the time of export: a moved page's entire history carries its final
title, and namespace derives from that title. Redirect status, by
contrast, is determined per revision from each revision's own text: the
`revision_is_redirect` and `revision_redirect_target` columns record
whether a revision's text begins with a redirect directive and where it
points. `#REDIRECT` is recognized on every wiki; wikis that also use a
localized keyword can add it with `--redirect-aliases` (for example,
`--redirect-aliases OMDIRIGERING` for Swedish wikis).

The most commonly useful options (`wikiq --help` describes them all):

- `-n ID` limits output to a namespace, and can be given more than once.
  `-rr N` sets how many prior edits are checked when detecting reverts.
- `--collapse-user` collapses each sequence of consecutive edits by the
  same user into a single row, which can address problems with text
  persistence measures.
- `-p [METHOD]` computes content persistence measures for each revision:
  persistent token revisions, tokens added, and tokens removed. The
  available methods are `sequence` (the default), `segment` (robust to
  content moves but slower), `wikidiff2` (like segment, using the
  wikidiff2 diff engine), and `legacy` (the behavior of older research
  projects). Persistence is the slowest thing wikiq computes.
- `-d` outputs a structured diff for each revision; `-t` outputs the
  full revision text.
- `--external-links`, `--citations`, `--wikilinks`, `--templates`, and
  `--headings` parse each revision's wikitext and add a column with the
  extracted elements.
- `-RP REGEX -RPl LABEL` searches revision text for a regular expression
  and reports the matches in a column named by the label; repeat the
  pair for multiple patterns. `-CP`/`-CPl` do the same for edit
  summaries.
- `--resume` continues an interrupted run from the last complete line of
  an existing JSONL output file. Combined with `--time-limit HOURS`,
  this supports processing very large dumps in bounded chunks.
- `--print-schema` prints a Spark-compatible JSON schema for the
  configured output and exits.

## Tests

From the repository root:

    pip install pytest pandas pytest-asyncio pytest-benchmark
    pytest test/

Run the suite from the root—some tests open fixture files by paths
relative to it. The full suite processes several real dumps from
`test/dumps/`; expected outputs live in `test/baseline_output/`.

Tests covering `--diff`, `-p wikidiff2`, and `-p legacy` skip when their
optional dependency is absent, so a base install reports skips rather
than failures.

## Authors

wikiq is written and maintained by members of the [Community Data
Science Collective](https://wiki.communitydata.science/). Contributors
include Benjamin Mako Hill, Nathan TeBlunthuis, Will Beason, Sohyeon
Hwang, and Kaylea Champion. It builds on earlier versions of the tool
written in Python and in C++ by Benjamin Mako Hill.

## License

wikiq is free software: you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free
Software Foundation, either version 3 of the License, or (at your
option) any later version. See [COPYING](COPYING) for the full text.