Prepare wikiq for release: fixes, dependency cleanup, and packaging #2

Open
mako wants to merge 33 commits from release_review into jsonl-output
Showing only changes of commit ff1f42dae5 - Show all commits

View File

@@ -19,8 +19,26 @@ this repository:
[uv](https://docs.astral.sh/uv/) works too (`uv sync`) and is convenient [uv](https://docs.astral.sh/uv/) works too (`uv sync`) and is convenient
for development, but it is not required. for development, but it is not required.
One dependency, `pywikidiff2`, is installed from source and compiles a This installs everything needed for the columns most people use, and
C++ extension, so a C++ compiler must be available at install time. requires no compiler. Two features carry heavier dependencies and are
therefore optional.
`--diff` and `-p wikidiff2` need
[pywikidiff2](https://gitea.communitydata.science/groceryheist/pywikidiff2),
a Python binding for MediaWiki's wikidiff2 diff engine. It is not on
PyPI and compiles a C++ extension, so it needs a C++ compiler and
libthai:
pip install 'pywikidiff2 @ git+https://gitea.communitydata.science/groceryheist/pywikidiff2.git'
`-p legacy` needs `mediawiki-utilities`, which is only useful for
reproducing results from older research projects:
pip install '.[legacy]'
wikiq tells you which of these to install if you use an option that
needs one. The other persistence methods, including the default
`-p sequence`, need neither.
Wikimedia dumps are usually compressed as 7z (most common), gz, or bz2. Wikimedia dumps are usually compressed as 7z (most common), gz, or bz2.
wikiq reads these by running your system's decompression tools, so it wikiq reads these by running your system's decompression tools, so it
@@ -80,12 +98,16 @@ The most commonly useful options (`wikiq --help` describes them all):
From the repository root: From the repository root:
uv run pytest test/ pip install pytest pandas pytest-asyncio pytest-benchmark
pytest test/
Run the suite from the root—some tests open fixture files by paths Run the suite from the root—some tests open fixture files by paths
relative to it. The full suite processes several real dumps from relative to it. The full suite processes several real dumps from
`test/dumps/` and takes a couple of minutes; expected outputs live in `test/dumps/`; expected outputs live in `test/baseline_output/`.
`test/baseline_output/`.
Tests covering `--diff`, `-p wikidiff2`, and `-p legacy` skip when their
optional dependency is absent, so a base install reports skips rather
than failures.
## Authors ## Authors