Prepare wikiq for release: fixes, dependency cleanup, and packaging #2

Open
mako wants to merge 33 commits from release_review into jsonl-output
Showing only changes of commit ff1f42dae5 - Show all commits

View File

@@ -19,8 +19,26 @@ this repository:
[uv](https://docs.astral.sh/uv/) works too (`uv sync`) and is convenient
for development, but it is not required.
One dependency, `pywikidiff2`, is installed from source and compiles a
C++ extension, so a C++ compiler must be available at install time.
This installs everything needed for the columns most people use, and
requires no compiler. Two features carry heavier dependencies and are
therefore optional.
`--diff` and `-p wikidiff2` need
[pywikidiff2](https://gitea.communitydata.science/groceryheist/pywikidiff2),
a Python binding for MediaWiki's wikidiff2 diff engine. It is not on
PyPI and compiles a C++ extension, so it needs a C++ compiler and
libthai:
pip install 'pywikidiff2 @ git+https://gitea.communitydata.science/groceryheist/pywikidiff2.git'
`-p legacy` needs `mediawiki-utilities`, which is only useful for
reproducing results from older research projects:
pip install '.[legacy]'
wikiq tells you which of these to install if you use an option that
needs one. The other persistence methods, including the default
`-p sequence`, need neither.
Wikimedia dumps are usually compressed as 7z (most common), gz, or bz2.
wikiq reads these by running your system's decompression tools, so it
@@ -80,12 +98,16 @@ The most commonly useful options (`wikiq --help` describes them all):
From the repository root:
uv run pytest test/
pip install pytest pandas pytest-asyncio pytest-benchmark
pytest test/
Run the suite from the root—some tests open fixture files by paths
relative to it. The full suite processes several real dumps from
`test/dumps/` and takes a couple of minutes; expected outputs live in
`test/baseline_output/`.
`test/dumps/`; expected outputs live in `test/baseline_output/`.
Tests covering `--diff`, `-p wikidiff2`, and `-p legacy` skip when their
optional dependency is absent, so a base install reports skips rather
than failures.
## Authors