collective/wikiq
20
0
Fork 0
Code Issues Pull Requests 3 Activity

put the base install ahead of the optional features in the README

Browse Source 
The installation section led with the two optional dependencies and left
the decompression tools until last, which put the awkward parts first
and buried something every run actually needs. Describe the base install
and the compression handling together, then the optional features under
their own headings.

Drop the note about uv. It said only that uv also works, which anyone
who uses uv already knows.

Also open with what wikiq is rather than describing the repository as a
collection of tools, now that the repository is named after the program.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Benjamin Mako Hill
2026-08-13 16:15:55 -07:00
parent 1b55acb01f
commit 0fa8b3d9ba
1 changed files with 21 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files

40
README.md
View File

@@ -1,11 +1,10 @@
# wikiq
Tools for converting MediaWiki XML database dumps—the "history" dumps
that include every revision of every page—into tabular datasets for
research. The main tool is `wikiq`, a command line program that produces
one row per revision with metadata such as the page, namespace,
timestamp, editor, text size, and revert status, plus a range of
optional computed columns.
wikiq converts MediaWiki XML database dumps—the "history" dumps that
include every revision of every page—into tabular datasets for research.
It is a command line program that produces one row per revision with
metadata such as the page, namespace, timestamp, editor, text size, and
revert status, plus a range of optional computed columns.
## Installation
@@ -16,12 +15,17 @@ this repository:
cd wikiq
pip install .
[uv](https://docs.astral.sh/uv/) works too (`uv sync`) and is convenient
for development, but it is not required.
Wikimedia dumps are usually compressed as 7z (most common), gz, or bz2.
wikiq reads these by running your system's decompression tools, so it
depends on `7za`, `zcat`, and `bzcat` for those respective formats. On
Debian or Ubuntu, `apt install 7zip` provides `7za`; the others are
standard.
This installs everything needed for the columns most people use, and
requires no compiler. Two features carry heavier dependencies and are
therefore optional.
That is everything most uses need, and none of it requires a compiler.
Two further features carry heavier dependencies, so they are kept out of
the base install.
### Diffs and wikidiff2 persistence
`--diff` and `-p wikidiff2` need
[pywikidiff2](https://gitea.communitydata.science/groceryheist/pywikidiff2),
@@ -31,20 +35,18 @@ libthai:
pip install 'pywikidiff2 @ git+https://gitea.communitydata.science/groceryheist/pywikidiff2.git'
The other persistence methods, including the default `-p sequence`, do
not need it.
### Legacy persistence
`-p legacy` needs `mediawiki-utilities`, which is only useful for
reproducing results from older research projects:
pip install '.[legacy]'
wikiq tells you which of these to install if you use an option that
needs one. The other persistence methods, including the default
`-p sequence`, need neither.
Wikimedia dumps are usually compressed as 7z (most common), gz, or bz2.
wikiq reads these by running your system's decompression tools, so it
depends on `7za`, `zcat`, and `bzcat` for those respective formats. On
Debian or Ubuntu, `apt install 7zip` provides `7za`; the others are
standard.
needs one.
## Usage