add --content-sizes: split revisions into content and markup

Wikitravel-style guides get written twice. Someone who does not know
wikitext contributes the information:

    Wonderful Hotel, +15555550123

and someone who does comes along later and polishes it:

    {{listing|name=Wonderful Hotel|phone=+15555550123}}

Both carry the same information. The second adds two dozen characters of
markup around it. --content-sizes measures that split, emitting two
nullable int64 columns from a single mwparserfromhell tree walk per
revision.

content_chars counts what a contributor who did not know wikitext would
have typed: text nodes, template parameter values, tag contents,
wikilink labels (or targets when unlabeled), external link URLs and
their labels, and heading titles. markup_chars is everything else --
braces, pipes, parameter names, tag names and attributes, link targets,
quote marks, and comments, which never reach a reader.

The two columns sum to the revision length, so their ratio reads
directly as how much of a revision is formatting. That invariant is what
the end-to-end test checks on a real dump, along with nulls for deleted
revisions, which come through the same machinery as the other
parser-based columns.

Emphasis needs no special handling: mwparserfromhell represents ''' and
'' as b and i tags, so the quote marks fall out as markup and the words
they wrap as content, which is the wanted answer.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-06 18:09:29 -07:00
parent 57988849cc
commit af62c4cea4
5 changed files with 229 additions and 2 deletions

View File

@@ -85,6 +85,13 @@ The most commonly useful options (`wikiq --help` describes them all):
- `--external-links`, `--citations`, `--wikilinks`, `--templates`, and
`--headings` parse each revision's wikitext and add a column with the
extracted elements.
- `--content-sizes` splits each revision into `content_chars`, the
information it carries, and `markup_chars`, the wikitext around it. A
contributor who does not know wikitext writes `Wonderful Hotel,
+15555550123`; someone polishing it later writes
`{{listing|name=Wonderful Hotel|phone=+15555550123}}`. Both carry the
same information, and the second adds two dozen characters of markup.
The two columns sum to the revision's length.
- `-RP REGEX -RPl LABEL` searches revision text for a regular expression
and reports the matches in a column named by the label; repeat the
pair for multiple patterns. `-CP`/`-CPl` do the same for edit