Prepare wikiq for release: fixes, dependency cleanup, and packaging #2

Open
mako wants to merge 33 commits from release_review into jsonl-output
Showing only changes of commit 0fa8b3d9ba - Show all commits

View File

@@ -1,11 +1,10 @@
# wikiq # wikiq
Tools for converting MediaWiki XML database dumps—the "history" dumps wikiq converts MediaWiki XML database dumps—the "history" dumps that
that include every revision of every page—into tabular datasets for include every revision of every page—into tabular datasets for research.
research. The main tool is `wikiq`, a command line program that produces It is a command line program that produces one row per revision with
one row per revision with metadata such as the page, namespace, metadata such as the page, namespace, timestamp, editor, text size, and
timestamp, editor, text size, and revert status, plus a range of revert status, plus a range of optional computed columns.
optional computed columns.
## Installation ## Installation
@@ -16,12 +15,17 @@ this repository:
cd wikiq cd wikiq
pip install . pip install .
[uv](https://docs.astral.sh/uv/) works too (`uv sync`) and is convenient Wikimedia dumps are usually compressed as 7z (most common), gz, or bz2.
for development, but it is not required. wikiq reads these by running your system's decompression tools, so it
depends on `7za`, `zcat`, and `bzcat` for those respective formats. On
Debian or Ubuntu, `apt install 7zip` provides `7za`; the others are
standard.
This installs everything needed for the columns most people use, and That is everything most uses need, and none of it requires a compiler.
requires no compiler. Two features carry heavier dependencies and are Two further features carry heavier dependencies, so they are kept out of
therefore optional. the base install.
### Diffs and wikidiff2 persistence
`--diff` and `-p wikidiff2` need `--diff` and `-p wikidiff2` need
[pywikidiff2](https://gitea.communitydata.science/groceryheist/pywikidiff2), [pywikidiff2](https://gitea.communitydata.science/groceryheist/pywikidiff2),
@@ -31,20 +35,18 @@ libthai:
pip install 'pywikidiff2 @ git+https://gitea.communitydata.science/groceryheist/pywikidiff2.git' pip install 'pywikidiff2 @ git+https://gitea.communitydata.science/groceryheist/pywikidiff2.git'
The other persistence methods, including the default `-p sequence`, do
not need it.
### Legacy persistence
`-p legacy` needs `mediawiki-utilities`, which is only useful for `-p legacy` needs `mediawiki-utilities`, which is only useful for
reproducing results from older research projects: reproducing results from older research projects:
pip install '.[legacy]' pip install '.[legacy]'
wikiq tells you which of these to install if you use an option that wikiq tells you which of these to install if you use an option that
needs one. The other persistence methods, including the default needs one.
`-p sequence`, need neither.
Wikimedia dumps are usually compressed as 7z (most common), gz, or bz2.
wikiq reads these by running your system's decompression tools, so it
depends on `7za`, `zcat`, and `bzcat` for those respective formats. On
Debian or Ubuntu, `apt install 7zip` provides `7za`; the others are
standard.
## Usage ## Usage