Python 3.9 reached end of life in October 2025, and 3.11 is where the
current ecosystem sits: it unlocks pyarrow 25 (we were held at 21, the
last release supporting 3.9), more-itertools 11, and pandas 3 for the
test suite, along with the large CPython 3.11 interpreter speedups on
exactly the kind of CPU-bound work wikiq does. Nothing in the code
needed changes.
Raising the floor exposed that PyPI's yamlconf 0.2.6 (a transitive
dependency via deltas) pins pyyaml == 5.4.1, which has no wheels for
3.11+ and no longer builds from source under current setuptools. Add a
PEP 508 reference to the fork that loosens the pin; it can be dropped
when https://github.com/halfak/yamlconf/pull/10 is merged and released.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drop the [tool.uv.sources] pins of deltas, mwxml, and yamlconf to forks
under a personal github account:
- The deltas fork was byte-identical to upstream.
- The mwxml fork carried one patch to mwxml.map(), which wikiq does not
call, and was missing eleven upstream commits of fixes. Require
mwxml >= 0.3.8 for those fixes; note that 0.3.7 changed namespace
handling to trust the dump's embedded <ns> tag rather than overriding
it based on the title prefix.
- The yamlconf fork loosened an old pyyaml pin in requirements.txt.
yamlconf is not imported by wikiq and is only needed transitively by
deltas, so drop the direct dependency; the PyPI release installs fine,
though its pyyaml pin is old (worth fixing upstream).
With pywikidiff2 already declared as a PEP 508 direct https reference,
no uv-specific configuration remains and the package installs with pip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Move pywikidiff2 from a [tool.uv.sources] entry to a direct git URL in
the dependency list, and switch it from ssh to anonymous https. The ssh
URL required an account on the gitea instance just to install the
package. A PEP 508 direct reference is understood by pip and every other
standards-compliant installer, not just uv, so this is a step toward
making uv an optional convenience rather than a requirement for working
with this repository.
Hatchling rejects direct references by default because they are not
allowed in packages published to PyPI (this one is not); set
tool.hatch.metadata.allow-direct-references to permit them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The spark indexing pass is moving out of this repository, and
src/wikiq_spark was never committed here: pyproject.toml declared a
wikiq-spark console script and wheel package that do not exist in the
tree, and pulled in pyspark for every install of a tool that never
imports it. The --print-schema flag and its Spark-format schema
converters remain, since they are pure Python and produce the schema
the external indexing pass consumes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Before, only fandom wikis dumps were compressed with .7z.
These archives can have several .xml files in the .7z; not just one.
So we need to have a flag for the fandom-2020 dumps.
This fixes the bug so .7z archives work in either case.
