add --content-sizes: split revisions into content and markup
Wikitravel-style guides get written twice. Someone who does not know
wikitext contributes the information:
Wonderful Hotel, +15555550123
and someone who does comes along later and polishes it:
{{listing|name=Wonderful Hotel|phone=+15555550123}}
Both carry the same information. The second adds two dozen characters of
markup around it. --content-sizes measures that split, emitting two
nullable int64 columns from a single mwparserfromhell tree walk per
revision.
content_chars counts what a contributor who did not know wikitext would
have typed: text nodes, template parameter values, tag contents,
wikilink labels (or targets when unlabeled), external link URLs and
their labels, and heading titles. markup_chars is everything else --
braces, pipes, parameter names, tag names and attributes, link targets,
quote marks, and comments, which never reach a reader.
The two columns sum to the revision length, so their ratio reads
directly as how much of a revision is formatting. That invariant is what
the end-to-end test checks on a real dump, along with nulls for deleted
revisions, which come through the same machinery as the other
parser-based columns.
Emphasis needs no special handling: mwparserfromhell represents ''' and
'' as b and i tags, so the quote marks fall out as markup and the words
they wrap as content, which is the wanted answer.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -85,6 +85,13 @@ The most commonly useful options (`wikiq --help` describes them all):
|
||||
- `--external-links`, `--citations`, `--wikilinks`, `--templates`, and
|
||||
`--headings` parse each revision's wikitext and add a column with the
|
||||
extracted elements.
|
||||
- `--content-sizes` splits each revision into `content_chars`, the
|
||||
information it carries, and `markup_chars`, the wikitext around it. A
|
||||
contributor who does not know wikitext writes `Wonderful Hotel,
|
||||
+15555550123`; someone polishing it later writes
|
||||
`{{listing|name=Wonderful Hotel|phone=+15555550123}}`. Both carry the
|
||||
same information, and the second adds two dozen characters of markup.
|
||||
The two columns sum to the revision's length.
|
||||
- `-RP REGEX -RPl LABEL` searches revision text for a regular expression
|
||||
and reports the matches in a column named by the label; repeat the
|
||||
pair for multiple patterns. `-CP`/`-CPl` do the same for edit
|
||||
|
||||
Reference in New Issue
Block a user