make redirect detection revision-level

The redirect_target column added in 59fea19 was page-level: the page's
state at dump time, stamped onto every revision row of the page. That
invited the wrong inference that a given revision was a redirect, and
the column was empty for dumps whose export never wrote <redirect>
elements, such as the 2023 wikitravel.org scrapes; a Swedish Wikitravel
scrape with 732 in-text redirect revisions produced no redirect signal
at all.

Remove that column and detect redirects from each revision's own text
instead. revision_is_redirect records whether the text begins with a
redirect directive and revision_redirect_target records the directive's
link target with any fragment and label stripped. #REDIRECT is
recognized on every wiki; localized keywords (e.g. OMDIRIGERING on
Swedish wikis) can be added with --redirect-aliases. Revisions with
deleted or unavailable text get nulls in both columns. The redirect-map
use case behind 59fea19 survives: the last revision's
revision_redirect_target per page reconstructs the page-level map, now
also on dumps without <redirect> elements.

Document that title and namespace are page-level identity values as of
the time of export, not historical facts about each revision.

Regenerate the test baselines for the column change. Every regenerated
file was verified to differ from its predecessor only by removing
redirect_target and adding the two new columns, with identical values
in all shared columns.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-06 23:41:57 -07:00
parent c1ee926211
commit 40132f04dd
22 changed files with 137498 additions and 137340 deletions

View File

@@ -37,6 +37,16 @@ Each output row describes one revision. The output format follows the
else produces tab-separated values. With no dump file argument, wikiq
reads XML on stdin and writes to stdout.
The `title` and `namespace` columns are page-level identity values as
of the time of export: a moved page's entire history carries its final
title, and namespace derives from that title. Redirect status, by
contrast, is determined per revision from each revision's own text: the
`revision_is_redirect` and `revision_redirect_target` columns record
whether a revision's text begins with a redirect directive and where it
points. `#REDIRECT` is recognized on every wiki; wikis that also use a
localized keyword can add it with `--redirect-aliases` (for example,
`--redirect-aliases OMDIRIGERING` for Swedish wikis).
The most commonly useful options (`wikiq --help` describes them all):
- `-n ID` limits output to a namespace, and can be given more than once.