put the base install ahead of the optional features in the README

The installation section led with the two optional dependencies and left
the decompression tools until last, which put the awkward parts first
and buried something every run actually needs. Describe the base install
and the compression handling together, then the optional features under
their own headings.

Drop the note about uv. It said only that uv also works, which anyone
who uses uv already knows.

Also open with what wikiq is rather than describing the repository as a
collection of tools, now that the repository is named after the program.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-13 16:15:55 -07:00
parent 1b55acb01f
commit 0fa8b3d9ba

View File

@@ -1,11 +1,10 @@
# wikiq # wikiq
Tools for converting MediaWiki XML database dumps—the "history" dumps wikiq converts MediaWiki XML database dumps—the "history" dumps that
that include every revision of every page—into tabular datasets for include every revision of every page—into tabular datasets for research.
research. The main tool is `wikiq`, a command line program that produces It is a command line program that produces one row per revision with
one row per revision with metadata such as the page, namespace, metadata such as the page, namespace, timestamp, editor, text size, and
timestamp, editor, text size, and revert status, plus a range of revert status, plus a range of optional computed columns.
optional computed columns.
## Installation ## Installation
@@ -16,12 +15,17 @@ this repository:
cd wikiq cd wikiq
pip install . pip install .
[uv](https://docs.astral.sh/uv/) works too (`uv sync`) and is convenient Wikimedia dumps are usually compressed as 7z (most common), gz, or bz2.
for development, but it is not required. wikiq reads these by running your system's decompression tools, so it
depends on `7za`, `zcat`, and `bzcat` for those respective formats. On
Debian or Ubuntu, `apt install 7zip` provides `7za`; the others are
standard.
This installs everything needed for the columns most people use, and That is everything most uses need, and none of it requires a compiler.
requires no compiler. Two features carry heavier dependencies and are Two further features carry heavier dependencies, so they are kept out of
therefore optional. the base install.
### Diffs and wikidiff2 persistence
`--diff` and `-p wikidiff2` need `--diff` and `-p wikidiff2` need
[pywikidiff2](https://gitea.communitydata.science/groceryheist/pywikidiff2), [pywikidiff2](https://gitea.communitydata.science/groceryheist/pywikidiff2),
@@ -31,20 +35,18 @@ libthai:
pip install 'pywikidiff2 @ git+https://gitea.communitydata.science/groceryheist/pywikidiff2.git' pip install 'pywikidiff2 @ git+https://gitea.communitydata.science/groceryheist/pywikidiff2.git'
The other persistence methods, including the default `-p sequence`, do
not need it.
### Legacy persistence
`-p legacy` needs `mediawiki-utilities`, which is only useful for `-p legacy` needs `mediawiki-utilities`, which is only useful for
reproducing results from older research projects: reproducing results from older research projects:
pip install '.[legacy]' pip install '.[legacy]'
wikiq tells you which of these to install if you use an option that wikiq tells you which of these to install if you use an option that
needs one. The other persistence methods, including the default needs one.
`-p sequence`, need neither.
Wikimedia dumps are usually compressed as 7z (most common), gz, or bz2.
wikiq reads these by running your system's decompression tools, so it
depends on `7za`, `zcat`, and `bzcat` for those respective formats. On
Debian or Ubuntu, `apt install 7zip` provides `7za`; the others are
standard.
## Usage ## Usage