Prepare wikiq for release: fixes, dependency cleanup, and packaging #2

Open
mako wants to merge 33 commits from release_review into jsonl-output
Showing only changes of commit 0fa8b3d9ba - Show all commits

View File

@@ -1,11 +1,10 @@
# wikiq
Tools for converting MediaWiki XML database dumps—the "history" dumps
that include every revision of every page—into tabular datasets for
research. The main tool is `wikiq`, a command line program that produces
one row per revision with metadata such as the page, namespace,
timestamp, editor, text size, and revert status, plus a range of
optional computed columns.
wikiq converts MediaWiki XML database dumps—the "history" dumps that
include every revision of every page—into tabular datasets for research.
It is a command line program that produces one row per revision with
metadata such as the page, namespace, timestamp, editor, text size, and
revert status, plus a range of optional computed columns.
## Installation
@@ -16,12 +15,17 @@ this repository:
cd wikiq
pip install .
[uv](https://docs.astral.sh/uv/) works too (`uv sync`) and is convenient
for development, but it is not required.
Wikimedia dumps are usually compressed as 7z (most common), gz, or bz2.
wikiq reads these by running your system's decompression tools, so it
depends on `7za`, `zcat`, and `bzcat` for those respective formats. On
Debian or Ubuntu, `apt install 7zip` provides `7za`; the others are
standard.
This installs everything needed for the columns most people use, and
requires no compiler. Two features carry heavier dependencies and are
therefore optional.
That is everything most uses need, and none of it requires a compiler.
Two further features carry heavier dependencies, so they are kept out of
the base install.
### Diffs and wikidiff2 persistence
`--diff` and `-p wikidiff2` need
[pywikidiff2](https://gitea.communitydata.science/groceryheist/pywikidiff2),
@@ -31,20 +35,18 @@ libthai:
pip install 'pywikidiff2 @ git+https://gitea.communitydata.science/groceryheist/pywikidiff2.git'
The other persistence methods, including the default `-p sequence`, do
not need it.
### Legacy persistence
`-p legacy` needs `mediawiki-utilities`, which is only useful for
reproducing results from older research projects:
pip install '.[legacy]'
wikiq tells you which of these to install if you use an option that
needs one. The other persistence methods, including the default
`-p sequence`, need neither.
Wikimedia dumps are usually compressed as 7z (most common), gz, or bz2.
wikiq reads these by running your system's decompression tools, so it
depends on `7za`, `zcat`, and `bzcat` for those respective formats. On
Debian or Ubuntu, `apt install 7zip` provides `7za`; the others are
standard.
needs one.
## Usage