add --content-sizes: decompose revision content into prose and structured characters
A single mwparserfromhell tree walk per revision emits two nullable int64 columns. prose_chars counts rendered content outside any template or tag: text nodes, wikilink labels (or targets when unlabeled), external link labels, and heading titles. structured_chars counts the same kinds of content inside template parameter values or tag contents, with each character assigned by its nearest enclosing container. Markup syntax, template and tag names, parameter names, attributes, bare URLs, and HTML comments count in neither, so syntax overhead is derivable as the total size minus the two columns. Deleted revisions and parse timeouts yield nulls through the same machinery as the other parser-based columns and do not change which revisions are processed. Note that mwparserfromhell parses bold and italic markup as tags, so emphasized text counts as structured. Includes hand-computed fixture tests for the counting rules and an end-to-end test on a dump with deleted revisions verifying nulls and the invariant prose_chars + structured_chars <= revision size. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -55,6 +55,10 @@ The most commonly useful options (`wikiq --help` describes them all):
|
||||
- `--external-links`, `--citations`, `--wikilinks`, `--templates`, and
|
||||
`--headings` parse each revision's wikitext and add a column with the
|
||||
extracted elements.
|
||||
- `--content-sizes` counts each revision's rendered content characters
|
||||
in two columns: `prose_chars` (content outside templates and tags) and
|
||||
`structured_chars` (template parameter values and tag inner text).
|
||||
Markup syntax and comments count in neither.
|
||||
- `-RP REGEX -RPl LABEL` searches revision text for a regular expression
|
||||
and reports the matches in a column named by the label; repeat the
|
||||
pair for multiple patterns. `-CP`/`-CPl` do the same for edit
|
||||
|
||||
Reference in New Issue
Block a user