diff --git a/README.md b/README.md index ed7aaef..be04d19 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,10 @@ # wikiq -Tools for converting MediaWiki XML database dumps—the "history" dumps -that include every revision of every page—into tabular datasets for -research. The main tool is `wikiq`, a command line program that produces -one row per revision with metadata such as the page, namespace, -timestamp, editor, text size, and revert status, plus a range of -optional computed columns. +wikiq converts MediaWiki XML database dumps—the "history" dumps that +include every revision of every page—into tabular datasets for research. +It is a command line program that produces one row per revision with +metadata such as the page, namespace, timestamp, editor, text size, and +revert status, plus a range of optional computed columns. ## Installation @@ -16,12 +15,17 @@ this repository: cd wikiq pip install . -[uv](https://docs.astral.sh/uv/) works too (`uv sync`) and is convenient -for development, but it is not required. +Wikimedia dumps are usually compressed as 7z (most common), gz, or bz2. +wikiq reads these by running your system's decompression tools, so it +depends on `7za`, `zcat`, and `bzcat` for those respective formats. On +Debian or Ubuntu, `apt install 7zip` provides `7za`; the others are +standard. -This installs everything needed for the columns most people use, and -requires no compiler. Two features carry heavier dependencies and are -therefore optional. +That is everything most uses need, and none of it requires a compiler. +Two further features carry heavier dependencies, so they are kept out of +the base install. + +### Diffs and wikidiff2 persistence `--diff` and `-p wikidiff2` need [pywikidiff2](https://gitea.communitydata.science/groceryheist/pywikidiff2), @@ -31,20 +35,18 @@ libthai: pip install 'pywikidiff2 @ git+https://gitea.communitydata.science/groceryheist/pywikidiff2.git' +The other persistence methods, including the default `-p sequence`, do +not need it. + +### Legacy persistence + `-p legacy` needs `mediawiki-utilities`, which is only useful for reproducing results from older research projects: pip install '.[legacy]' wikiq tells you which of these to install if you use an option that -needs one. The other persistence methods, including the default -`-p sequence`, need neither. - -Wikimedia dumps are usually compressed as 7z (most common), gz, or bz2. -wikiq reads these by running your system's decompression tools, so it -depends on `7za`, `zcat`, and `bzcat` for those respective formats. On -Debian or Ubuntu, `apt install 7zip` provides `7za`; the others are -standard. +needs one. ## Usage