Benjamin Mako Hill
0b6e0fbc70
Port the regex counting feature from the mako_changes-20230429 branch (2ff4d60): the -RPc/--revision-pattern-count and -CPc/--comment-pattern-count flags cause revision and comment patterns to output the number of matches instead of the matched text, with 0 (never null) for revisions with no matches or deleted content. The flags apply to all revision or comment patterns respectively, and count columns are typed int64 in the output schema. Counting the matches of common or large patterns previously required returning the full matched text, which made for very large outputs. Also take the code review suggestion from that branch (933ca75): use assignment expressions so matching no longer makes redundant calls to search() before finditer()/findall() or calls match.group() twice. Includes unit tests for the count semantics and schema types, and an end-to-end test that verifies counts against the number of matches recomputed from each revision's text and comment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When you install this from git, you will need to first clone the repository:
git clone git://projects.mako.cc/mediawiki_dump_tools
From within the repository working directory, initiatlize and set up the submodule like:
git submodule init
git submodule update
Wikimedia dumps are usually in a compressed format such as 7z (most common), gz, or bz2. Wikiq uses your computer's compression software to read these files. Therefore wikiq depends on [7za]{.title-ref}, [gzcat]{.title-ref}, and [zcat]{.title-ref}.
Dependencies
These non-Python dependencies must be installed on your system for wikiq and its associated tests to work.
- 7zip
- ffmpeg
A new diff engine based on [_wikidiff2]{.title-ref} can be used for word-persistence. Wikiq can also output the diffs between each page revision. This requires installing Wikidiff 2 on your system. On Debian or Ubuntu Linux this can be done via.
apt-get install php-wikidiff2
You may have to also run. sudo phpenmod wikidiff2.
Tests ----To run tests:
python -m unittest test.Wikiq_Unit_Test
TODO:
- [] Output metadata about the run. What parameters were used? What versions of deltas?
- [] Url encoding by default