put the base install ahead of the optional features in the README
The installation section led with the two optional dependencies and left the decompression tools until last, which put the awkward parts first and buried something every run actually needs. Describe the base install and the compression handling together, then the optional features under their own headings. Drop the note about uv. It said only that uv also works, which anyone who uses uv already knows. Also open with what wikiq is rather than describing the repository as a collection of tools, now that the repository is named after the program. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
40
README.md
40
README.md
@@ -1,11 +1,10 @@
|
|||||||
# wikiq
|
# wikiq
|
||||||
|
|
||||||
Tools for converting MediaWiki XML database dumps—the "history" dumps
|
wikiq converts MediaWiki XML database dumps—the "history" dumps that
|
||||||
that include every revision of every page—into tabular datasets for
|
include every revision of every page—into tabular datasets for research.
|
||||||
research. The main tool is `wikiq`, a command line program that produces
|
It is a command line program that produces one row per revision with
|
||||||
one row per revision with metadata such as the page, namespace,
|
metadata such as the page, namespace, timestamp, editor, text size, and
|
||||||
timestamp, editor, text size, and revert status, plus a range of
|
revert status, plus a range of optional computed columns.
|
||||||
optional computed columns.
|
|
||||||
|
|
||||||
## Installation
|
## Installation
|
||||||
|
|
||||||
@@ -16,12 +15,17 @@ this repository:
|
|||||||
cd wikiq
|
cd wikiq
|
||||||
pip install .
|
pip install .
|
||||||
|
|
||||||
[uv](https://docs.astral.sh/uv/) works too (`uv sync`) and is convenient
|
Wikimedia dumps are usually compressed as 7z (most common), gz, or bz2.
|
||||||
for development, but it is not required.
|
wikiq reads these by running your system's decompression tools, so it
|
||||||
|
depends on `7za`, `zcat`, and `bzcat` for those respective formats. On
|
||||||
|
Debian or Ubuntu, `apt install 7zip` provides `7za`; the others are
|
||||||
|
standard.
|
||||||
|
|
||||||
This installs everything needed for the columns most people use, and
|
That is everything most uses need, and none of it requires a compiler.
|
||||||
requires no compiler. Two features carry heavier dependencies and are
|
Two further features carry heavier dependencies, so they are kept out of
|
||||||
therefore optional.
|
the base install.
|
||||||
|
|
||||||
|
### Diffs and wikidiff2 persistence
|
||||||
|
|
||||||
`--diff` and `-p wikidiff2` need
|
`--diff` and `-p wikidiff2` need
|
||||||
[pywikidiff2](https://gitea.communitydata.science/groceryheist/pywikidiff2),
|
[pywikidiff2](https://gitea.communitydata.science/groceryheist/pywikidiff2),
|
||||||
@@ -31,20 +35,18 @@ libthai:
|
|||||||
|
|
||||||
pip install 'pywikidiff2 @ git+https://gitea.communitydata.science/groceryheist/pywikidiff2.git'
|
pip install 'pywikidiff2 @ git+https://gitea.communitydata.science/groceryheist/pywikidiff2.git'
|
||||||
|
|
||||||
|
The other persistence methods, including the default `-p sequence`, do
|
||||||
|
not need it.
|
||||||
|
|
||||||
|
### Legacy persistence
|
||||||
|
|
||||||
`-p legacy` needs `mediawiki-utilities`, which is only useful for
|
`-p legacy` needs `mediawiki-utilities`, which is only useful for
|
||||||
reproducing results from older research projects:
|
reproducing results from older research projects:
|
||||||
|
|
||||||
pip install '.[legacy]'
|
pip install '.[legacy]'
|
||||||
|
|
||||||
wikiq tells you which of these to install if you use an option that
|
wikiq tells you which of these to install if you use an option that
|
||||||
needs one. The other persistence methods, including the default
|
needs one.
|
||||||
`-p sequence`, need neither.
|
|
||||||
|
|
||||||
Wikimedia dumps are usually compressed as 7z (most common), gz, or bz2.
|
|
||||||
wikiq reads these by running your system's decompression tools, so it
|
|
||||||
depends on `7za`, `zcat`, and `bzcat` for those respective formats. On
|
|
||||||
Debian or Ubuntu, `apt install 7zip` provides `7za`; the others are
|
|
||||||
standard.
|
|
||||||
|
|
||||||
## Usage
|
## Usage
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user