A single mwparserfromhell tree walk per revision emits two nullable
int64 columns. prose_chars counts rendered content outside any template
or tag: text nodes, wikilink labels (or targets when unlabeled),
external link labels, and heading titles. structured_chars counts the
same kinds of content inside template parameter values or tag contents,
with each character assigned by its nearest enclosing container. Markup
syntax, template and tag names, parameter names, attributes, bare URLs,
and HTML comments count in neither, so syntax overhead is derivable as
the total size minus the two columns.
Deleted revisions and parse timeouts yield nulls through the same
machinery as the other parser-based columns and do not change which
revisions are processed. Note that mwparserfromhell parses bold and
italic markup as tags, so emphasized text counts as structured.
Includes hand-computed fixture tests for the counting rules and an
end-to-end test on a dump with deleted revisions verifying nulls and
the invariant prose_chars + structured_chars <= revision size.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Describe what wikiq is and rewrite the installation, usage, and test
instructions to match the current code: pip or uv installation from the
gitea repository, the Python 3.11 requirement, output format selection
by extension, and the current option set including persistence methods,
wikitext extraction, regex matching and counting, and JSONL resume. Add
an authors section crediting the Community Data Science Collective
contributors and the earlier Python and C++ versions of the tool.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add the full license text as COPYING (the same text Debian ships in
common-licenses), declare the license and license file in
pyproject.toml, add copyright and permission notices to the source
files, and describe the license in the README. Also replace the
placeholder package description in pyproject.toml.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the auto-converted README.md (which carried pandoc title-ref
spans and a garbled Tests heading from a malformed rst underline) with
a clean conversion of README.rst, and remove the rst version.
pyproject.toml already declares README.md as the package readme.
Content is unchanged; bringing it up to date is left for a separate
change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
