make mediawiki-utilities an optional dependency

-p legacy is the only thing that uses mediawiki-utilities, and it exists
solely to reproduce numbers from research predating the mwxml and
mwtypes split. The package is a 2015 monolith that also contains
mw.database and mw.api, so it declares requests and pymysql
unconditionally: every wikiq install pulled a MySQL client and six other
packages to support one flag, building them from an sdist that has no
wheel.

Move it to a `legacy` extra. mw.lib.persistence imports nothing from
mw.database or mw.api, so nothing about -p legacy changes for people who
install the extra, and everyone else stops paying for it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
(cherry picked from commit 66dcc6d8cea620a619fc19121b5c27a085b11991)
This commit is contained in:
2026-08-13 11:58:12 -07:00
parent 5725c66040
commit 70b0ebcf0c
2 changed files with 30 additions and 2 deletions

View File

@@ -67,6 +67,22 @@ def require_pywikidiff2(feature: str):
return pywikidiff2
def require_mw_persistence():
"""Import and return mw.lib.persistence, or exit explaining how to get it."""
try:
from mw.lib import persistence
except ImportError:
raise SystemExit(
"-p legacy requires mediawiki-utilities, which wikiq no longer "
"installs by default.\n"
"Install it with:\n"
" pip install 'mediawiki-dump-tools[legacy]'\n"
"This method exists to reproduce results from older research "
"projects; -p sequence is the current equivalent."
)
return persistence
def pyarrow_type_to_spark(pa_type):
"""Convert a PyArrow type to Spark JSON schema format."""
if pa.types.is_int64(pa_type):
@@ -845,7 +861,7 @@ class WikiqParser:
wikidiff_matcher, revert_radius=PERSISTENCE_RADIUS
)
else:
from mw.lib import persistence
persistence = require_mw_persistence()
persist_state = persistence.State()
# Pending persistence values waiting for window to fill
@@ -1298,6 +1314,8 @@ def main():
require_pywikidiff2("--diff")
if persist == PersistMethod.wikidiff2:
require_pywikidiff2("-p wikidiff2")
elif persist == PersistMethod.legacy:
require_mw_persistence()
if args.namespace_filter is not None:
namespaces = args.namespace_filter