Prepare wikiq for release: fixes, dependency cleanup, and packaging #2
40
README.md
40
README.md
@@ -1,11 +1,10 @@
|
||||
# wikiq
|
||||
|
||||
Tools for converting MediaWiki XML database dumps—the "history" dumps
|
||||
that include every revision of every page—into tabular datasets for
|
||||
research. The main tool is `wikiq`, a command line program that produces
|
||||
one row per revision with metadata such as the page, namespace,
|
||||
timestamp, editor, text size, and revert status, plus a range of
|
||||
optional computed columns.
|
||||
wikiq converts MediaWiki XML database dumps—the "history" dumps that
|
||||
include every revision of every page—into tabular datasets for research.
|
||||
It is a command line program that produces one row per revision with
|
||||
metadata such as the page, namespace, timestamp, editor, text size, and
|
||||
revert status, plus a range of optional computed columns.
|
||||
|
||||
## Installation
|
||||
|
||||
@@ -16,12 +15,17 @@ this repository:
|
||||
cd wikiq
|
||||
pip install .
|
||||
|
||||
[uv](https://docs.astral.sh/uv/) works too (`uv sync`) and is convenient
|
||||
for development, but it is not required.
|
||||
Wikimedia dumps are usually compressed as 7z (most common), gz, or bz2.
|
||||
wikiq reads these by running your system's decompression tools, so it
|
||||
depends on `7za`, `zcat`, and `bzcat` for those respective formats. On
|
||||
Debian or Ubuntu, `apt install 7zip` provides `7za`; the others are
|
||||
standard.
|
||||
|
||||
This installs everything needed for the columns most people use, and
|
||||
requires no compiler. Two features carry heavier dependencies and are
|
||||
therefore optional.
|
||||
That is everything most uses need, and none of it requires a compiler.
|
||||
Two further features carry heavier dependencies, so they are kept out of
|
||||
the base install.
|
||||
|
||||
### Diffs and wikidiff2 persistence
|
||||
|
||||
`--diff` and `-p wikidiff2` need
|
||||
[pywikidiff2](https://gitea.communitydata.science/groceryheist/pywikidiff2),
|
||||
@@ -31,20 +35,18 @@ libthai:
|
||||
|
||||
pip install 'pywikidiff2 @ git+https://gitea.communitydata.science/groceryheist/pywikidiff2.git'
|
||||
|
||||
The other persistence methods, including the default `-p sequence`, do
|
||||
not need it.
|
||||
|
||||
### Legacy persistence
|
||||
|
||||
`-p legacy` needs `mediawiki-utilities`, which is only useful for
|
||||
reproducing results from older research projects:
|
||||
|
||||
pip install '.[legacy]'
|
||||
|
||||
wikiq tells you which of these to install if you use an option that
|
||||
needs one. The other persistence methods, including the default
|
||||
`-p sequence`, need neither.
|
||||
|
||||
Wikimedia dumps are usually compressed as 7z (most common), gz, or bz2.
|
||||
wikiq reads these by running your system's decompression tools, so it
|
||||
depends on `7za`, `zcat`, and `bzcat` for those respective formats. On
|
||||
Debian or Ubuntu, `apt install 7zip` provides `7za`; the others are
|
||||
standard.
|
||||
needs one.
|
||||
|
||||
## Usage
|
||||
|
||||
|
||||
Reference in New Issue
Block a user