pip install . copies the source, so a branch switch leaves the installed
console script running the old code. Say so, since that is easy to miss.
Setup on klone differs enough that duplicating it here would go stale;
link to the collective's wiki rather than restating it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The suite completes in about ninety seconds, not fifteen minutes. The
old figure discouraged running it as part of an ordinary change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
On klone the venv is created inside the repo with create_cdsc_venv, so
it otherwise shows up as untracked in every git status.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Brings in revision-level redirect detection and its regenerated
baselines. Conflicts were the expected adjacent insertions where both
branches extended the same seams (build_table and WikiqParser
signatures, the argparse block, the build_table call sites, and the
test file); resolved by keeping both branches' additions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The redirect_target column added in 59fea19 was page-level: the page's
state at dump time, stamped onto every revision row of the page. That
invited the wrong inference that a given revision was a redirect, and
the column was empty for dumps whose export never wrote <redirect>
elements, such as the 2023 wikitravel.org scrapes; a Swedish Wikitravel
scrape with 732 in-text redirect revisions produced no redirect signal
at all.
Remove that column and detect redirects from each revision's own text
instead. revision_is_redirect records whether the text begins with a
redirect directive and revision_redirect_target records the directive's
link target with any fragment and label stripped. #REDIRECT is
recognized on every wiki; localized keywords (e.g. OMDIRIGERING on
Swedish wikis) can be added with --redirect-aliases. Revisions with
deleted or unavailable text get nulls in both columns. The redirect-map
use case behind 59fea19 survives: the last revision's
revision_redirect_target per page reconstructs the page-level map, now
also on dumps without <redirect> elements.
Document that title and namespace are page-level identity values as of
the time of export, not historical facts about each revision.
Regenerate the test baselines for the column change. Every regenerated
file was verified to differ from its predecessor only by removing
redirect_target and adding the two new columns, with identical values
in all shared columns.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A single mwparserfromhell tree walk per revision emits two nullable
int64 columns. prose_chars counts rendered content outside any template
or tag: text nodes, wikilink labels (or targets when unlabeled),
external link labels, and heading titles. structured_chars counts the
same kinds of content inside template parameter values or tag contents,
with each character assigned by its nearest enclosing container. Markup
syntax, template and tag names, parameter names, attributes, bare URLs,
and HTML comments count in neither, so syntax overhead is derivable as
the total size minus the two columns.
Deleted revisions and parse timeouts yield nulls through the same
machinery as the other parser-based columns and do not change which
revisions are processed. Note that mwparserfromhell parses bold and
italic markup as tags, so emphasized text counts as structured.
Includes hand-computed fixture tests for the counting rules and an
end-to-end test on a dump with deleted revisions verifying nulls and
the invariant prose_chars + structured_chars <= revision size.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resolve the package version from installed metadata, report it via a
--version flag, and print it at the start of each run so output logs
record which wikiq produced them. Bump the version to 0.2.0 for the
changes on this branch: regex match counting, the removal of parquet
output, the Python 3.11 floor, and the dependency cleanup.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Describe what wikiq is and rewrite the installation, usage, and test
instructions to match the current code: pip or uv installation from the
gitea repository, the Python 3.11 requirement, output format selection
by extension, and the current option set including persistence methods,
wikitext extraction, regex matching and counting, and JSONL resume. Add
an authors section crediting the Community Data Science Collective
contributors and the earlier Python and C++ versions of the tool.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add the full license text as COPYING (the same text Debian ships in
common-licenses), declare the license and license file in
pyproject.toml, add copyright and permission notices to the source
files, and describe the license in the README. Also replace the
placeholder package description in pyproject.toml.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
No test reads this file; the regex tests use the basic_regextest and
capturegroup_regextest baselines.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the auto-converted README.md (which carried pandoc title-ref
spans and a garbled Tests heading from a malformed rst underline) with
a clean conversion of README.rst, and remove the rst version.
pyproject.toml already declares README.md as the package readme.
Content is unchanged; bringing it up to date is left for a separate
change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
assert_equal_enough wrote its inputs to files named "token" and "rev"
in the current directory on every invocation, littering the repository
root whenever the test suite ran.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Python 3.9 reached end of life in October 2025, and 3.11 is where the
current ecosystem sits: it unlocks pyarrow 25 (we were held at 21, the
last release supporting 3.9), more-itertools 11, and pandas 3 for the
test suite, along with the large CPython 3.11 interpreter speedups on
exactly the kind of CPU-bound work wikiq does. Nothing in the code
needed changes.
Raising the floor exposed that PyPI's yamlconf 0.2.6 (a transitive
dependency via deltas) pins pyyaml == 5.4.1, which has no wheels for
3.11+ and no longer builds from source under current setuptools. Add a
PEP 508 reference to the fork that loosens the pin; it can be dropped
when https://github.com/halfak/yamlconf/pull/10 is merged and released.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drop the [tool.uv.sources] pins of deltas, mwxml, and yamlconf to forks
under a personal github account:
- The deltas fork was byte-identical to upstream.
- The mwxml fork carried one patch to mwxml.map(), which wikiq does not
call, and was missing eleven upstream commits of fixes. Require
mwxml >= 0.3.8 for those fixes; note that 0.3.7 changed namespace
handling to trust the dump's embedded <ns> tag rather than overriding
it based on the title prefix.
- The yamlconf fork loosened an old pyyaml pin in requirements.txt.
yamlconf is not imported by wikiq and is only needed transitively by
deltas, so drop the direct dependency; the PyPI release installs fine,
though its pyyaml pin is old (worth fixing upstream).
With pywikidiff2 already declared as a PEP 508 direct https reference,
no uv-specific configuration remains and the package installs with pip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
WikiqPage unconditionally prepended the namespace name to titles of
pages outside the main namespace. mwxml >= 0.3.7 keeps the namespace
prefix on the title when the dump carries <ns> tags (it previously
stripped it), which made wikiq emit double-prefixed titles like
"Category:Category:Alaska". Check for the prefix before adding it.
mwxml still strips the prefix on its fallback path for dumps without
<ns> tags, so those titles are still prefixed as before.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Move pywikidiff2 from a [tool.uv.sources] entry to a direct git URL in
the dependency list, and switch it from ssh to anonymous https. The ssh
URL required an account on the gitea instance just to install the
package. A PEP 508 direct reference is understood by pip and every other
standards-compliant installer, not just uv, so this is a step toward
making uv an optional convenience rather than a requirement for working
with this repository.
Hatchling rejects direct references by default because they are not
allowed in packages published to PyPI (this one is not); set
tool.hatch.metadata.allow-direct-references to permit them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The spark indexing pass is moving out of this repository, and
src/wikiq_spark was never committed here: pyproject.toml declared a
wikiq-spark console script and wheel package that do not exist in the
tree, and pulled in pyspark for every install of a tool that never
imports it. The --print-schema flag and its Spark-format schema
converters remain, since they are pure Python and produce the schema
the external indexing pass consumes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Port the regex counting feature from the mako_changes-20230429 branch
(2ff4d60): the -RPc/--revision-pattern-count and
-CPc/--comment-pattern-count flags cause revision and comment patterns
to output the number of matches instead of the matched text, with 0
(never null) for revisions with no matches or deleted content. The
flags apply to all revision or comment patterns respectively, and count
columns are typed int64 in the output schema.
Counting the matches of common or large patterns previously required
returning the full matched text, which made for very large outputs.
Also take the code review suggestion from that branch (933ca75): use
assignment expressions so matching no longer makes redundant calls to
search() before finditer()/findall() or calls match.group() twice.
Includes unit tests for the count semantics and schema types, and an
end-to-end test that verifies counts against the number of matches
recomputed from each revision's text and comment.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resume with collapsed revision groups was untested: the resume point is
the (articleid, revid) of the last written row, which for collapsed
output is the last revision of a group, and nothing verified that the
replay reconstructs group boundaries and collapsed_revs counts
identically across the resume point. Run sailormoon with
--collapse-user, truncate the output at the midpoint, resume, and
assert the result is identical to an uninterrupted run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test_infobox reads test/test_diff_revisions/test_infobox_from and
test_infobox_to, which were never committed to the repository, so the
test always failed with FileNotFoundError. Skip it when the fixtures
are missing so it runs again if they are ever added.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Commit 59fea19 added a redirect_target column to wikiq output but did
not regenerate the test baselines, leaving 14 baseline-comparison tests
failing. Regenerate the affected baselines from current output. Each
regenerated file was verified to differ from its old baseline only by
the addition of the new column: row counts and all values in shared
columns are identical.
Also add the noargs_sailormoon.jsonl baseline used by test_jsonl_noargs,
which was never committed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
build_table returns (table, reverts_column, wikitext_parser) but the
jsonl test helper still unpacked two values, so test_jsonl_noargs and
test_jsonl_tsv_equivalence failed with a ValueError before reading any
output. Also update the docstring, which still described the two-value
return.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Nate has given up on getting wikiq to output parquet directly because it
makes too many unpredictable large memory allocations. The workflow now
outputs JSONL in a single pass and then uses spark (wikiq_spark) to index
it as parquet in a second pass.
Remove the parquet output path along with the machinery that existed only
to support it: checkpoint files, resume temp-file merging, namespace
partitioning, and file rotation (--partition-namespaces,
--max-revisions-per-file). Resume support remains for JSONL output, where
the resume point is derived from the last complete line of the output
file. Also remove the parquet tests and baseline files.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RegexPair.matchmake crashed with a TypeError when a capture-group
pattern was applied to a revision whose text or comment was deleted or
suppressed (content is None). Guard both matching paths against None,
and make the no-capture-group path emit a None column for such
revisions instead of omitting the key entirely.
This carries forward the fix Kaylea Champion and Mako Hill made on the
mako_changes-20230429 branch (7e6cd5b), which predated the rewrite.
Adds a unit test for matchmake(None) and an end-to-end test against the
ikwiki dump, which contains revisions with deleted text and comments.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per-page adaptive parse skip: once mwparserfromhell times out on a
revision of size N bytes, force-timeout future revisions on the same
page whose text is >= N bytes instead of waiting another 60s on each
one. State resets per page, so well-behaved pages are unaffected.
Motivated by zhwiki bot-vandalism log pages (e.g. ns=4
'Wikipedia:当前的破坏/...' at 4.6GB across 6133 revisions) where every
revision triggers the 60s SIGALRM, blocking the worker for hours.
