Add --content-sizes #4

mako · 2026-08-14T00:58:32Z

mako commented

2026-08-14 00:58:32 +00:00

Adds --content-sizes, which splits each revision into content_chars
-- the information it carries -- and markup_chars, the wikitext around
it.

A travel guide gets written twice. Someone who does not know wikitext
contributes Wonderful Hotel, +15555550123; someone who does comes
along later and writes
{{listing|name=Wonderful Hotel|phone=+15555550123}}. Both carry the
same information; the second adds two dozen characters of markup. This
measures that split, so the ratio reads directly as how much of a
revision is formatting rather than information.

The two columns sum to the revision length, which the end-to-end test
checks against a real dump.

One commit. Suite: 67 passed, 2 skipped.

After PR 1 merges, retarget this to jsonl-output.

Adds `--content-sizes`, which splits each revision into `content_chars` -- the information it carries -- and `markup_chars`, the wikitext around it. A travel guide gets written twice. Someone who does not know wikitext contributes `Wonderful Hotel, +15555550123`; someone who does comes along later and writes `{{listing|name=Wonderful Hotel|phone=+15555550123}}`. Both carry the same information; the second adds two dozen characters of markup. This measures that split, so the ratio reads directly as how much of a revision is formatting rather than information. The two columns sum to the revision length, which the end-to-end test checks against a real dump. One commit. Suite: 67 passed, 2 skipped. After PR 1 merges, retarget this to `jsonl-output`.

mako added 1 commit 2026-08-14 00:58:32 +00:00

add --content-sizes: split revisions into content and markup af62c4cea4

Wikitravel-style guides get written twice. Someone who does not know
wikitext contributes the information:

    Wonderful Hotel, +15555550123

and someone who does comes along later and polishes it:

    {{listing|name=Wonderful Hotel|phone=+15555550123}}

Both carry the same information. The second adds two dozen characters of
markup around it. --content-sizes measures that split, emitting two
nullable int64 columns from a single mwparserfromhell tree walk per
revision.

content_chars counts what a contributor who did not know wikitext would
have typed: text nodes, template parameter values, tag contents,
wikilink labels (or targets when unlabeled), external link URLs and
their labels, and heading titles. markup_chars is everything else --
braces, pipes, parameter names, tag names and attributes, link targets,
quote marks, and comments, which never reach a reader.

The two columns sum to the revision length, so their ratio reads
directly as how much of a revision is formatting. That invariant is what
the end-to-end test checks on a real dump, along with nulls for deleted
revisions, which come through the same machinery as the other
parser-based columns.

Emphasis needs no special handling: mwparserfromhell represents ''' and
'' as b and i tags, so the quote marks fall out as markup and the words
they wrap as content, which is the wanted answer.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

mako referenced this pull request

2026-08-14 00:58:56 +00:00

Prepare wikiq for release: fixes, dependency cleanup, and packaging #2

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin content-sizes:content-sizes

git checkout content-sizes

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: collective/wikiq#4