add --content-sizes: split revisions into content and markup

Wikitravel-style guides get written twice. Someone who does not know
wikitext contributes the information:

    Wonderful Hotel, +15555550123

and someone who does comes along later and polishes it:

    {{listing|name=Wonderful Hotel|phone=+15555550123}}

Both carry the same information. The second adds two dozen characters of
markup around it. --content-sizes measures that split, emitting two
nullable int64 columns from a single mwparserfromhell tree walk per
revision.

content_chars counts what a contributor who did not know wikitext would
have typed: text nodes, template parameter values, tag contents,
wikilink labels (or targets when unlabeled), external link URLs and
their labels, and heading titles. markup_chars is everything else --
braces, pipes, parameter names, tag names and attributes, link targets,
quote marks, and comments, which never reach a reader.

The two columns sum to the revision length, so their ratio reads
directly as how much of a revision is formatting. That invariant is what
the end-to-end test checks on a real dump, along with nulls for deleted
revisions, which come through the same machinery as the other
parser-based columns.

Emphasis needs no special handling: mwparserfromhell represents ''' and
'' as b and i tags, so the quote marks fall out as markup and the words
they wrap as content, which is the wanted answer.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

.gitignore

@@ -4,12 +4,14 @@
 *.xml.xz
 *.swp
 
-# Lockfiles
-uv.lock
-
 # JetBrains
 /.idea
 
+/.claude/settings.local.json*
+
 # Python build and test output
 __pycache__/
 /test/test_output/
+
+# Virtualenv (created with create_cdsc_venv on klone)
+/.venv/

.gitmodules

@@ -1,3 +0,0 @@
-[submodule "mediawiki-php-wikidiff2"]
-	path = mediawiki-php-wikidiff2
-	url = https://github.com/wikimedia/mediawiki-php-wikidiff2/

.python-version

@@ -1 +1 @@
-3.9
+3.11

CHANGELOG.md

@@ -0,0 +1,73 @@
+# Changelog
+
+## 0.3.0
+
+Changes relative to what people have been running. Earlier version tags
+in this repository were internal markers rather than releases, so there
+is no previous release to compare against.
+
+### Incompatible changes
+
+- **Redirect detection is now per revision.** The page-level
+  `redirect_target` column has been replaced by `revision_is_redirect`
+  and `revision_redirect_target`, which classify each revision from its
+  own text. The old column reported a page's redirect status as of the
+  time the dump was exported and applied that single value to every
+  revision in the page's history, which is wrong for any page that was
+  turned into a redirect, or turned back, at some point in its life.
+  Revisions with deleted text report null in both columns rather than
+  false. Wikis using a localized redirect keyword can add it with
+  `--redirect-aliases`; `#REDIRECT` is always recognized.
+
+  Code reading `redirect_target` needs updating.
+
+- **Parquet output has been removed**, along with the
+  `--partition-namespaces` and `--max-revisions-per-file` options that
+  only applied to it. Use JSONL, which supports `--resume`.
+
+- **pyspark is no longer a dependency** and the `wikiq-spark` entry point
+  is gone. The second pass of that pipeline was never in this
+  repository.
+
+- **Python 3.11 or later is required.** Python 3.9 reached end of life in
+  October 2025, and the floor unlocks current pyarrow, more-itertools and
+  pandas, along with the CPython 3.11 speedups on the CPU-bound work
+  wikiq does.
+
+- **The project is now called wikiq**, rather than mediawiki_dump_tools.
+
+### Installation
+
+- **wikiq installs with plain `pip` and needs no C++ compiler.**
+  Previously every install required pywikidiff2, built from an ssh-only
+  git URL, so anyone outside the collective could not install the tool at
+  all. It is now optional and needed only for `--diff` and
+  `-p wikidiff2`.
+
+- **`-p legacy` needs the new `legacy` extra.** It was the only user of
+  `mediawiki-utilities`, a 2015 package that also pulls in a MySQL client
+  and six other packages. Everyone was paying for that to support one
+  flag.
+
+- **Dependencies resolve from PyPI** rather than from pinned forks under
+  personal accounts. Requires mwxml >= 0.3.8, which fixes eleven upstream
+  issues; note that mwxml 0.3.7 changed namespace handling to trust the
+  dump's embedded `<ns>` tag.
+
+### Added
+
+- `--version`, and the version is printed to stderr on every run so it
+  lands in job logs beside the output.
+- Licensing: wikiq is GPL-3.0-or-later. See `COPYING`.
+
+### Fixed
+
+- Regex matching (`-RP`/`-CP`) crashed with a `TypeError` on revisions
+  whose text or comment was deleted or suppressed. Deleted content now
+  reports null.
+- Page titles in non-main namespaces were prefixed twice under mwxml
+  0.3.7 and later, producing titles like `Category:Category:Alaska`.
+- The test suite had 16 failing tests, and `pytest test/` silently
+  collected only half of it because the main test file matched neither of
+  pytest's discovery patterns. It is now `test_wikiq.py` and the suite
+  runs green.

COPYING

@@ -0,0 +1,674 @@
+                    GNU GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+                            Preamble
+
+  The GNU General Public License is a free, copyleft license for
+software and other kinds of works.
+
+  The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works.  By contrast,
+the GNU General Public License is intended to guarantee your freedom to
+share and change all versions of a program--to make sure it remains free
+software for all its users.  We, the Free Software Foundation, use the
+GNU General Public License for most of our software; it applies also to
+any other work released this way by its authors.  You can apply it to
+your programs, too.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+them if you wish), that you receive source code or can get it if you
+want it, that you can change the software or use pieces of it in new
+free programs, and that you know you can do these things.
+
+  To protect your rights, we need to prevent others from denying you
+these rights or asking you to surrender the rights.  Therefore, you have
+certain responsibilities if you distribute copies of the software, or if
+you modify it: responsibilities to respect the freedom of others.
+
+  For example, if you distribute copies of such a program, whether
+gratis or for a fee, you must pass on to the recipients the same
+freedoms that you received.  You must make sure that they, too, receive
+or can get the source code.  And you must show them these terms so they
+know their rights.
+
+  Developers that use the GNU GPL protect your rights with two steps:
+(1) assert copyright on the software, and (2) offer you this License
+giving you legal permission to copy, distribute and/or modify it.
+
+  For the developers' and authors' protection, the GPL clearly explains
+that there is no warranty for this free software.  For both users' and
+authors' sake, the GPL requires that modified versions be marked as
+changed, so that their problems will not be attributed erroneously to
+authors of previous versions.
+
+  Some devices are designed to deny users access to install or run
+modified versions of the software inside them, although the manufacturer
+can do so.  This is fundamentally incompatible with the aim of
+protecting users' freedom to change the software.  The systematic
+pattern of such abuse occurs in the area of products for individuals to
+use, which is precisely where it is most unacceptable.  Therefore, we
+have designed this version of the GPL to prohibit the practice for those
+products.  If such problems arise substantially in other domains, we
+stand ready to extend this provision to those domains in future versions
+of the GPL, as needed to protect the freedom of users.
+
+  Finally, every program is threatened constantly by software patents.
+States should not allow patents to restrict development and use of
+software on general-purpose computers, but in those that do, we wish to
+avoid the special danger that patents applied to a free program could
+make it effectively proprietary.  To prevent this, the GPL assures that
+patents cannot be used to render the program non-free.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+                       TERMS AND CONDITIONS
+
+  0. Definitions.
+
+  "This License" refers to version 3 of the GNU General Public License.
+
+  "Copyright" also means copyright-like laws that apply to other kinds of
+works, such as semiconductor masks.
+
+  "The Program" refers to any copyrightable work licensed under this
+License.  Each licensee is addressed as "you".  "Licensees" and
+"recipients" may be individuals or organizations.
+
+  To "modify" a work means to copy from or adapt all or part of the work
+in a fashion requiring copyright permission, other than the making of an
+exact copy.  The resulting work is called a "modified version" of the
+earlier work or a work "based on" the earlier work.
+
+  A "covered work" means either the unmodified Program or a work based
+on the Program.
+
+  To "propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy.  Propagation includes copying,
+distribution (with or without modification), making available to the
+public, and in some countries other activities as well.
+
+  To "convey" a work means any kind of propagation that enables other
+parties to make or receive copies.  Mere interaction with a user through
+a computer network, with no transfer of a copy, is not conveying.
+
+  An interactive user interface displays "Appropriate Legal Notices"
+to the extent that it includes a convenient and prominently visible
+feature that (1) displays an appropriate copyright notice, and (2)
+tells the user that there is no warranty for the work (except to the
+extent that warranties are provided), that licensees may convey the
+work under this License, and how to view a copy of this License.  If
+the interface presents a list of user commands or options, such as a
+menu, a prominent item in the list meets this criterion.
+
+  1. Source Code.
+
+  The "source code" for a work means the preferred form of the work
+for making modifications to it.  "Object code" means any non-source
+form of a work.
+
+  A "Standard Interface" means an interface that either is an official
+standard defined by a recognized standards body, or, in the case of
+interfaces specified for a particular programming language, one that
+is widely used among developers working in that language.
+
+  The "System Libraries" of an executable work include anything, other
+than the work as a whole, that (a) is included in the normal form of
+packaging a Major Component, but which is not part of that Major
+Component, and (b) serves only to enable use of the work with that
+Major Component, or to implement a Standard Interface for which an
+implementation is available to the public in source code form.  A
+"Major Component", in this context, means a major essential component
+(kernel, window system, and so on) of the specific operating system
+(if any) on which the executable work runs, or a compiler used to
+produce the work, or an object code interpreter used to run it.
+
+  The "Corresponding Source" for a work in object code form means all
+the source code needed to generate, install, and (for an executable
+work) run the object code and to modify the work, including scripts to
+control those activities.  However, it does not include the work's
+System Libraries, or general-purpose tools or generally available free
+programs which are used unmodified in performing those activities but
+which are not part of the work.  For example, Corresponding Source
+includes interface definition files associated with source files for
+the work, and the source code for shared libraries and dynamically
+linked subprograms that the work is specifically designed to require,
+such as by intimate data communication or control flow between those
+subprograms and other parts of the work.
+
+  The Corresponding Source need not include anything that users
+can regenerate automatically from other parts of the Corresponding
+Source.
+
+  The Corresponding Source for a work in source code form is that
+same work.
+
+  2. Basic Permissions.
+
+  All rights granted under this License are granted for the term of
+copyright on the Program, and are irrevocable provided the stated
+conditions are met.  This License explicitly affirms your unlimited
+permission to run the unmodified Program.  The output from running a
+covered work is covered by this License only if the output, given its
+content, constitutes a covered work.  This License acknowledges your
+rights of fair use or other equivalent, as provided by copyright law.
+
+  You may make, run and propagate covered works that you do not
+convey, without conditions so long as your license otherwise remains
+in force.  You may convey covered works to others for the sole purpose
+of having them make modifications exclusively for you, or provide you
+with facilities for running those works, provided that you comply with
+the terms of this License in conveying all material for which you do
+not control copyright.  Those thus making or running the covered works
+for you must do so exclusively on your behalf, under your direction
+and control, on terms that prohibit them from making any copies of
+your copyrighted material outside their relationship with you.
+
+  Conveying under any other circumstances is permitted solely under
+the conditions stated below.  Sublicensing is not allowed; section 10
+makes it unnecessary.
+
+  3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+  No covered work shall be deemed part of an effective technological
+measure under any applicable law fulfilling obligations under article
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
+similar laws prohibiting or restricting circumvention of such
+measures.
+
+  When you convey a covered work, you waive any legal power to forbid
+circumvention of technological measures to the extent such circumvention
+is effected by exercising rights under this License with respect to
+the covered work, and you disclaim any intention to limit operation or
+modification of the work as a means of enforcing, against the work's
+users, your or third parties' legal rights to forbid circumvention of
+technological measures.
+
+  4. Conveying Verbatim Copies.
+
+  You may convey verbatim copies of the Program's source code as you
+receive it, in any medium, provided that you conspicuously and
+appropriately publish on each copy an appropriate copyright notice;
+keep intact all notices stating that this License and any
+non-permissive terms added in accord with section 7 apply to the code;
+keep intact all notices of the absence of any warranty; and give all
+recipients a copy of this License along with the Program.
+
+  You may charge any price or no price for each copy that you convey,
+and you may offer support or warranty protection for a fee.
+
+  5. Conveying Modified Source Versions.
+
+  You may convey a work based on the Program, or the modifications to
+produce it from the Program, in the form of source code under the
+terms of section 4, provided that you also meet all of these conditions:
+
+    a) The work must carry prominent notices stating that you modified
+    it, and giving a relevant date.
+
+    b) The work must carry prominent notices stating that it is
+    released under this License and any conditions added under section
+    7.  This requirement modifies the requirement in section 4 to
+    "keep intact all notices".
+
+    c) You must license the entire work, as a whole, under this
+    License to anyone who comes into possession of a copy.  This
+    License will therefore apply, along with any applicable section 7
+    additional terms, to the whole of the work, and all its parts,
+    regardless of how they are packaged.  This License gives no
+    permission to license the work in any other way, but it does not
+    invalidate such permission if you have separately received it.
+
+    d) If the work has interactive user interfaces, each must display
+    Appropriate Legal Notices; however, if the Program has interactive
+    interfaces that do not display Appropriate Legal Notices, your
+    work need not make them do so.
+
+  A compilation of a covered work with other separate and independent
+works, which are not by their nature extensions of the covered work,
+and which are not combined with it such as to form a larger program,
+in or on a volume of a storage or distribution medium, is called an
+"aggregate" if the compilation and its resulting copyright are not
+used to limit the access or legal rights of the compilation's users
+beyond what the individual works permit.  Inclusion of a covered work
+in an aggregate does not cause this License to apply to the other
+parts of the aggregate.
+
+  6. Conveying Non-Source Forms.
+
+  You may convey a covered work in object code form under the terms
+of sections 4 and 5, provided that you also convey the
+machine-readable Corresponding Source under the terms of this License,
+in one of these ways:
+
+    a) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by the
+    Corresponding Source fixed on a durable physical medium
+    customarily used for software interchange.
+
+    b) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by a
+    written offer, valid for at least three years and valid for as
+    long as you offer spare parts or customer support for that product
+    model, to give anyone who possesses the object code either (1) a
+    copy of the Corresponding Source for all the software in the
+    product that is covered by this License, on a durable physical
+    medium customarily used for software interchange, for a price no
+    more than your reasonable cost of physically performing this
+    conveying of source, or (2) access to copy the
+    Corresponding Source from a network server at no charge.
+
+    c) Convey individual copies of the object code with a copy of the
+    written offer to provide the Corresponding Source.  This
+    alternative is allowed only occasionally and noncommercially, and
+    only if you received the object code with such an offer, in accord
+    with subsection 6b.
+
+    d) Convey the object code by offering access from a designated
+    place (gratis or for a charge), and offer equivalent access to the
+    Corresponding Source in the same way through the same place at no
+    further charge.  You need not require recipients to copy the
+    Corresponding Source along with the object code.  If the place to
+    copy the object code is a network server, the Corresponding Source
+    may be on a different server (operated by you or a third party)
+    that supports equivalent copying facilities, provided you maintain
+    clear directions next to the object code saying where to find the
+    Corresponding Source.  Regardless of what server hosts the
+    Corresponding Source, you remain obligated to ensure that it is
+    available for as long as needed to satisfy these requirements.
+
+    e) Convey the object code using peer-to-peer transmission, provided
+    you inform other peers where the object code and Corresponding
+    Source of the work are being offered to the general public at no
+    charge under subsection 6d.
+
+  A separable portion of the object code, whose source code is excluded
+from the Corresponding Source as a System Library, need not be
+included in conveying the object code work.
+
+  A "User Product" is either (1) a "consumer product", which means any
+tangible personal property which is normally used for personal, family,
+or household purposes, or (2) anything designed or sold for incorporation
+into a dwelling.  In determining whether a product is a consumer product,
+doubtful cases shall be resolved in favor of coverage.  For a particular
+product received by a particular user, "normally used" refers to a
+typical or common use of that class of product, regardless of the status
+of the particular user or of the way in which the particular user
+actually uses, or expects or is expected to use, the product.  A product
+is a consumer product regardless of whether the product has substantial
+commercial, industrial or non-consumer uses, unless such uses represent
+the only significant mode of use of the product.
+
+  "Installation Information" for a User Product means any methods,
+procedures, authorization keys, or other information required to install
+and execute modified versions of a covered work in that User Product from
+a modified version of its Corresponding Source.  The information must
+suffice to ensure that the continued functioning of the modified object
+code is in no case prevented or interfered with solely because
+modification has been made.
+
+  If you convey an object code work under this section in, or with, or
+specifically for use in, a User Product, and the conveying occurs as
+part of a transaction in which the right of possession and use of the
+User Product is transferred to the recipient in perpetuity or for a
+fixed term (regardless of how the transaction is characterized), the
+Corresponding Source conveyed under this section must be accompanied
+by the Installation Information.  But this requirement does not apply
+if neither you nor any third party retains the ability to install
+modified object code on the User Product (for example, the work has
+been installed in ROM).
+
+  The requirement to provide Installation Information does not include a
+requirement to continue to provide support service, warranty, or updates
+for a work that has been modified or installed by the recipient, or for
+the User Product in which it has been modified or installed.  Access to a
+network may be denied when the modification itself materially and
+adversely affects the operation of the network or violates the rules and
+protocols for communication across the network.
+
+  Corresponding Source conveyed, and Installation Information provided,
+in accord with this section must be in a format that is publicly
+documented (and with an implementation available to the public in
+source code form), and must require no special password or key for
+unpacking, reading or copying.
+
+  7. Additional Terms.
+
+  "Additional permissions" are terms that supplement the terms of this
+License by making exceptions from one or more of its conditions.
+Additional permissions that are applicable to the entire Program shall
+be treated as though they were included in this License, to the extent
+that they are valid under applicable law.  If additional permissions
+apply only to part of the Program, that part may be used separately
+under those permissions, but the entire Program remains governed by
+this License without regard to the additional permissions.
+
+  When you convey a copy of a covered work, you may at your option
+remove any additional permissions from that copy, or from any part of
+it.  (Additional permissions may be written to require their own
+removal in certain cases when you modify the work.)  You may place
+additional permissions on material, added by you to a covered work,
+for which you have or can give appropriate copyright permission.
+
+  Notwithstanding any other provision of this License, for material you
+add to a covered work, you may (if authorized by the copyright holders of
+that material) supplement the terms of this License with terms:
+
+    a) Disclaiming warranty or limiting liability differently from the
+    terms of sections 15 and 16 of this License; or
+
+    b) Requiring preservation of specified reasonable legal notices or
+    author attributions in that material or in the Appropriate Legal
+    Notices displayed by works containing it; or
+
+    c) Prohibiting misrepresentation of the origin of that material, or
+    requiring that modified versions of such material be marked in
+    reasonable ways as different from the original version; or
+
+    d) Limiting the use for publicity purposes of names of licensors or
+    authors of the material; or
+
+    e) Declining to grant rights under trademark law for use of some
+    trade names, trademarks, or service marks; or
+
+    f) Requiring indemnification of licensors and authors of that
+    material by anyone who conveys the material (or modified versions of
+    it) with contractual assumptions of liability to the recipient, for
+    any liability that these contractual assumptions directly impose on
+    those licensors and authors.
+
+  All other non-permissive additional terms are considered "further
+restrictions" within the meaning of section 10.  If the Program as you
+received it, or any part of it, contains a notice stating that it is
+governed by this License along with a term that is a further
+restriction, you may remove that term.  If a license document contains
+a further restriction but permits relicensing or conveying under this
+License, you may add to a covered work material governed by the terms
+of that license document, provided that the further restriction does
+not survive such relicensing or conveying.
+
+  If you add terms to a covered work in accord with this section, you
+must place, in the relevant source files, a statement of the
+additional terms that apply to those files, or a notice indicating
+where to find the applicable terms.
+
+  Additional terms, permissive or non-permissive, may be stated in the
+form of a separately written license, or stated as exceptions;
+the above requirements apply either way.
+
+  8. Termination.
+
+  You may not propagate or modify a covered work except as expressly
+provided under this License.  Any attempt otherwise to propagate or
+modify it is void, and will automatically terminate your rights under
+this License (including any patent licenses granted under the third
+paragraph of section 11).
+
+  However, if you cease all violation of this License, then your
+license from a particular copyright holder is reinstated (a)
+provisionally, unless and until the copyright holder explicitly and
+finally terminates your license, and (b) permanently, if the copyright
+holder fails to notify you of the violation by some reasonable means
+prior to 60 days after the cessation.
+
+  Moreover, your license from a particular copyright holder is
+reinstated permanently if the copyright holder notifies you of the
+violation by some reasonable means, this is the first time you have
+received notice of violation of this License (for any work) from that
+copyright holder, and you cure the violation prior to 30 days after
+your receipt of the notice.
+
+  Termination of your rights under this section does not terminate the
+licenses of parties who have received copies or rights from you under
+this License.  If your rights have been terminated and not permanently
+reinstated, you do not qualify to receive new licenses for the same
+material under section 10.
+
+  9. Acceptance Not Required for Having Copies.
+
+  You are not required to accept this License in order to receive or
+run a copy of the Program.  Ancillary propagation of a covered work
+occurring solely as a consequence of using peer-to-peer transmission
+to receive a copy likewise does not require acceptance.  However,
+nothing other than this License grants you permission to propagate or
+modify any covered work.  These actions infringe copyright if you do
+not accept this License.  Therefore, by modifying or propagating a
+covered work, you indicate your acceptance of this License to do so.
+
+  10. Automatic Licensing of Downstream Recipients.
+
+  Each time you convey a covered work, the recipient automatically
+receives a license from the original licensors, to run, modify and
+propagate that work, subject to this License.  You are not responsible
+for enforcing compliance by third parties with this License.
+
+  An "entity transaction" is a transaction transferring control of an
+organization, or substantially all assets of one, or subdividing an
+organization, or merging organizations.  If propagation of a covered
+work results from an entity transaction, each party to that
+transaction who receives a copy of the work also receives whatever
+licenses to the work the party's predecessor in interest had or could
+give under the previous paragraph, plus a right to possession of the
+Corresponding Source of the work from the predecessor in interest, if
+the predecessor has it or can get it with reasonable efforts.
+
+  You may not impose any further restrictions on the exercise of the
+rights granted or affirmed under this License.  For example, you may
+not impose a license fee, royalty, or other charge for exercise of
+rights granted under this License, and you may not initiate litigation
+(including a cross-claim or counterclaim in a lawsuit) alleging that
+any patent claim is infringed by making, using, selling, offering for
+sale, or importing the Program or any portion of it.
+
+  11. Patents.
+
+  A "contributor" is a copyright holder who authorizes use under this
+License of the Program or a work on which the Program is based.  The
+work thus licensed is called the contributor's "contributor version".
+
+  A contributor's "essential patent claims" are all patent claims
+owned or controlled by the contributor, whether already acquired or
+hereafter acquired, that would be infringed by some manner, permitted
+by this License, of making, using, or selling its contributor version,
+but do not include claims that would be infringed only as a
+consequence of further modification of the contributor version.  For
+purposes of this definition, "control" includes the right to grant
+patent sublicenses in a manner consistent with the requirements of
+this License.
+
+  Each contributor grants you a non-exclusive, worldwide, royalty-free
+patent license under the contributor's essential patent claims, to
+make, use, sell, offer for sale, import and otherwise run, modify and
+propagate the contents of its contributor version.
+
+  In the following three paragraphs, a "patent license" is any express
+agreement or commitment, however denominated, not to enforce a patent
+(such as an express permission to practice a patent or covenant not to
+sue for patent infringement).  To "grant" such a patent license to a
+party means to make such an agreement or commitment not to enforce a
+patent against the party.
+
+  If you convey a covered work, knowingly relying on a patent license,
+and the Corresponding Source of the work is not available for anyone
+to copy, free of charge and under the terms of this License, through a
+publicly available network server or other readily accessible means,
+then you must either (1) cause the Corresponding Source to be so
+available, or (2) arrange to deprive yourself of the benefit of the
+patent license for this particular work, or (3) arrange, in a manner
+consistent with the requirements of this License, to extend the patent
+license to downstream recipients.  "Knowingly relying" means you have
+actual knowledge that, but for the patent license, your conveying the
+covered work in a country, or your recipient's use of the covered work
+in a country, would infringe one or more identifiable patents in that
+country that you have reason to believe are valid.
+
+  If, pursuant to or in connection with a single transaction or
+arrangement, you convey, or propagate by procuring conveyance of, a
+covered work, and grant a patent license to some of the parties
+receiving the covered work authorizing them to use, propagate, modify
+or convey a specific copy of the covered work, then the patent license
+you grant is automatically extended to all recipients of the covered
+work and works based on it.
+
+  A patent license is "discriminatory" if it does not include within
+the scope of its coverage, prohibits the exercise of, or is
+conditioned on the non-exercise of one or more of the rights that are
+specifically granted under this License.  You may not convey a covered
+work if you are a party to an arrangement with a third party that is
+in the business of distributing software, under which you make payment
+to the third party based on the extent of your activity of conveying
+the work, and under which the third party grants, to any of the
+parties who would receive the covered work from you, a discriminatory
+patent license (a) in connection with copies of the covered work
+conveyed by you (or copies made from those copies), or (b) primarily
+for and in connection with specific products or compilations that
+contain the covered work, unless you entered into that arrangement,
+or that patent license was granted, prior to 28 March 2007.
+
+  Nothing in this License shall be construed as excluding or limiting
+any implied license or other defenses to infringement that may
+otherwise be available to you under applicable patent law.
+
+  12. No Surrender of Others' Freedom.
+
+  If conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot convey a
+covered work so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you may
+not convey it at all.  For example, if you agree to terms that obligate you
+to collect a royalty for further conveying from those to whom you convey
+the Program, the only way you could satisfy both those terms and this
+License would be to refrain entirely from conveying the Program.
+
+  13. Use with the GNU Affero General Public License.
+
+  Notwithstanding any other provision of this License, you have
+permission to link or combine any covered work with a work licensed
+under version 3 of the GNU Affero General Public License into a single
+combined work, and to convey the resulting work.  The terms of this
+License will continue to apply to the part which is the covered work,
+but the special requirements of the GNU Affero General Public License,
+section 13, concerning interaction through a network will apply to the
+combination as such.
+
+  14. Revised Versions of this License.
+
+  The Free Software Foundation may publish revised and/or new versions of
+the GNU General Public License from time to time.  Such new versions will
+be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+  Each version is given a distinguishing version number.  If the
+Program specifies that a certain numbered version of the GNU General
+Public License "or any later version" applies to it, you have the
+option of following the terms and conditions either of that numbered
+version or of any later version published by the Free Software
+Foundation.  If the Program does not specify a version number of the
+GNU General Public License, you may choose any version ever published
+by the Free Software Foundation.
+
+  If the Program specifies that a proxy can decide which future
+versions of the GNU General Public License can be used, that proxy's
+public statement of acceptance of a version permanently authorizes you
+to choose that version for the Program.
+
+  Later license versions may give you additional or different
+permissions.  However, no additional obligations are imposed on any
+author or copyright holder as a result of your choosing to follow a
+later version.
+
+  15. Disclaimer of Warranty.
+
+  THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
+APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
+HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
+OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
+IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
+ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+  16. Limitation of Liability.
+
+  IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
+THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
+GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
+USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
+DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
+PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
+EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGES.
+
+  17. Interpretation of Sections 15 and 16.
+
+  If the disclaimer of warranty and limitation of liability provided
+above cannot be given local legal effect according to their terms,
+reviewing courts shall apply local law that most closely approximates
+an absolute waiver of all civil liability in connection with the
+Program, unless a warranty or assumption of liability accompanies a
+copy of the Program in return for a fee.
+
+                     END OF TERMS AND CONDITIONS
+
+            How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+state the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+Also add information on how to contact you by electronic and paper mail.
+
+  If the program does terminal interaction, make it output a short
+notice like this when it starts in an interactive mode:
+
+    <program>  Copyright (C) <year>  <name of author>
+    This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+    This is free software, and you are welcome to redistribute it
+    under certain conditions; type `show c' for details.
+
+The hypothetical commands `show w' and `show c' should show the appropriate
+parts of the General Public License.  Of course, your program's commands
+might be different; for a GUI interface, you would use an "about box".
+
+  You should also get your employer (if you work as a programmer) or school,
+if any, to sign a "copyright disclaimer" for the program, if necessary.
+For more information on this, and how to apply and follow the GNU GPL, see
+<https://www.gnu.org/licenses/>.
+
+  The GNU General Public License does not permit incorporating your program
+into proprietary programs.  If your program is a subroutine library, you
+may consider it more useful to permit linking proprietary applications with
+the library.  If this is what you want to do, use the GNU Lesser General
+Public License instead of this License.  But first, please read
+<https://www.gnu.org/licenses/why-not-lgpl.html>.

README.md

@@ -1,42 +1,133 @@
-When you install this from git, you will need to first clone the
-repository:
+# wikiq
 
-    git clone git://projects.mako.cc/mediawiki_dump_tools
+wikiq converts MediaWiki XML database dumps—the "history" dumps that
+include every revision of every page—into tabular datasets for research.
+It is a command line program that produces one row per revision with
+metadata such as the page, namespace, timestamp, editor, text size, and
+revert status, plus a range of optional computed columns.
 
-From within the repository working directory, initiatlize and set up the
-submodule like:
+## Installation
 
-    git submodule init
-    git submodule update
+wikiq requires Python 3.11 or later. Install it with pip from a clone of
+this repository:
 
-Wikimedia dumps are usually in a compressed format such as 7z (most
-common), gz, or bz2. Wikiq uses your computer\'s compression software to
-read these files. Therefore wikiq depends on [7za]{.title-ref},
-[gzcat]{.title-ref}, and [zcat]{.title-ref}.
+    git clone https://gitea.communitydata.science/collective/wikiq.git
+    cd wikiq
+    pip install .
 
-# Dependencies
+Wikimedia dumps are usually compressed as 7z (most common), gz, or bz2.
+wikiq reads these by running your system's decompression tools, so it
+depends on `7za`, `zcat`, and `bzcat` for those respective formats. On
+Debian or Ubuntu, `apt install 7zip` provides `7za`; the others are
+standard.
 
-These non-Python dependencies must be installed on your system for wikiq
-and its associated tests to work.
+That is everything most uses need, and none of it requires a compiler.
+Two further features carry heavier dependencies, so they are kept out of
+the base install.
 
--   7zip
--   ffmpeg
+### Diffs and wikidiff2 persistence
 
-A new diff engine based on [\_wikidiff2]{.title-ref} can be used for
-word-persistence. Wikiq can also output the diffs between each page
-revision. This requires installing Wikidiff 2 on your system. On Debian
-or Ubuntu Linux this can be done via.
+`--diff` and `-p wikidiff2` need
+[pywikidiff2](https://gitea.communitydata.science/groceryheist/pywikidiff2),
+a Python binding for MediaWiki's wikidiff2 diff engine. It is not on
+PyPI and compiles a C++ extension, so it needs a C++ compiler and
+libthai:
 
-`apt-get install php-wikidiff2`
+    pip install 'pywikidiff2 @ git+https://gitea.communitydata.science/groceryheist/pywikidiff2.git'
 
-You may have to also run. `sudo phpenmod wikidiff2`.
+The other persistence methods, including the default `-p sequence`, do
+not need it.
 
-Tests \-\-\--To run tests:
+### Legacy persistence
 
-    python -m unittest test.Wikiq_Unit_Test
+`-p legacy` needs `mediawiki-utilities`, which is only useful for
+reproducing results from older research projects:
 
-## TODO:
+    pip install '.[legacy]'
 
-1.  \[\] Output metadata about the run. What parameters were used? What
-    versions of deltas?
-2.  \[\] Url encoding by default
+wikiq tells you which of these to install if you use an option that
+needs one.
+
+## Usage
+
+    wikiq dump.xml.7z -o output/
+
+Each output row describes one revision. The output format follows the
+`-o` argument: a path ending in `.jsonl` produces JSON Lines, anything
+else produces tab-separated values. With no dump file argument, wikiq
+reads XML on stdin and writes to stdout.
+
+The `title` and `namespace` columns are page-level identity values as
+of the time of export: a moved page's entire history carries its final
+title, and namespace derives from that title. Redirect status, by
+contrast, is determined per revision from each revision's own text: the
+`revision_is_redirect` and `revision_redirect_target` columns record
+whether a revision's text begins with a redirect directive and where it
+points. `#REDIRECT` is recognized on every wiki; wikis that also use a
+localized keyword can add it with `--redirect-aliases` (for example,
+`--redirect-aliases OMDIRIGERING` for Swedish wikis).
+
+The most commonly useful options (`wikiq --help` describes them all):
+
+- `-n ID` limits output to a namespace, and can be given more than once.
+  `-rr N` sets how many prior edits are checked when detecting reverts.
+- `--collapse-user` collapses each sequence of consecutive edits by the
+  same user into a single row, which can address problems with text
+  persistence measures.
+- `-p [METHOD]` computes content persistence measures for each revision:
+  persistent token revisions, tokens added, and tokens removed. The
+  available methods are `sequence` (the default), `segment` (robust to
+  content moves but slower), `wikidiff2` (like segment, using the
+  wikidiff2 diff engine), and `legacy` (the behavior of older research
+  projects). Persistence is the slowest thing wikiq computes.
+- `-d` outputs a structured diff for each revision; `-t` outputs the
+  full revision text.
+- `--external-links`, `--citations`, `--wikilinks`, `--templates`, and
+  `--headings` parse each revision's wikitext and add a column with the
+  extracted elements.
+- `--content-sizes` splits each revision into `content_chars`, the
+  information it carries, and `markup_chars`, the wikitext around it. A
+  contributor who does not know wikitext writes `Wonderful Hotel,
+  +15555550123`; someone polishing it later writes
+  `{{listing|name=Wonderful Hotel|phone=+15555550123}}`. Both carry the
+  same information, and the second adds two dozen characters of markup.
+  The two columns sum to the revision's length.
+- `-RP REGEX -RPl LABEL` searches revision text for a regular expression
+  and reports the matches in a column named by the label; repeat the
+  pair for multiple patterns. `-CP`/`-CPl` do the same for edit
+  summaries.
+- `--resume` continues an interrupted run from the last complete line of
+  an existing JSONL output file. Combined with `--time-limit HOURS`,
+  this supports processing very large dumps in bounded chunks.
+- `--print-schema` prints a Spark-compatible JSON schema for the
+  configured output and exits.
+
+## Tests
+
+From the repository root:
+
+    pip install pytest pandas pytest-asyncio pytest-benchmark
+    pytest test/
+
+Run the suite from the root—some tests open fixture files by paths
+relative to it. The full suite processes several real dumps from
+`test/dumps/`; expected outputs live in `test/baseline_output/`.
+
+Tests covering `--diff`, `-p wikidiff2`, and `-p legacy` skip when their
+optional dependency is absent, so a base install reports skips rather
+than failures.
+
+## Authors
+
+wikiq is written and maintained by members of the [Community Data
+Science Collective](https://wiki.communitydata.science/). Contributors
+include Benjamin Mako Hill, Nathan TeBlunthuis, Will Beason, Sohyeon
+Hwang, and Kaylea Champion. It builds on earlier versions of the tool
+written in Python and in C++ by Benjamin Mako Hill.
+
+## License
+
+wikiq is free software: you can redistribute it and/or modify it under
+the terms of the GNU General Public License as published by the Free
+Software Foundation, either version 3 of the License, or (at your
+option) any later version. See [COPYING](COPYING) for the full text.

README.rst

@@ -1,41 +0,0 @@
-When you install this from git, you will need to first clone the repository::
-
-  git clone git://projects.mako.cc/mediawiki_dump_tools
-
-From within the repository working directory, initiatlize and set up the
-submodule like::
-
-  git submodule init
-  git submodule update
-
-
-Wikimedia dumps are usually in a compressed format such as 7z (most common), gz, or bz2. Wikiq uses your computer's compression software to read these files. Therefore wikiq depends on
-`7za`, `gzcat`, and `zcat`. 
-
-Dependencies
-----------------
-These non-Python dependencies must be installed on your system for wikiq and its
-associated tests to work.
-
-- 7zip
-- ffmpeg
-
-A new diff engine based on `_wikidiff2` can be used for word-persistence. Wikiq can also output the diffs between each page revision. This requires installing Wikidiff 2 on your system. On Debian or Ubuntu Linux this can be done via.
-
-``apt-get install php-wikidiff2``
-
-You may have to also run.
-``sudo phpenmod wikidiff2``.
-
-Tests
-----
-To run tests::
-
-   python -m unittest test.Wikiq_Unit_Test
-
-TODO:
-_______________
-1. [] Output metadata about the run. What parameters were used? What versions of deltas?
-2. [] Url encoding by default
-
-.. _wikidiff2: https://www.mediawiki.org/wiki/Wikidiff2

pyproject.toml

@@ -1,42 +1,90 @@
 [project]
-name = "mediawiki-dump-tools"
-version = "0.1.1"
-description = "Add your description here"
+name = "wikiq"
+version = "0.3.0"
+description = "Convert MediaWiki XML database dumps into tabular datasets for research"
 readme = "README.md"
-requires-python = ">=3.9"
+license = "GPL-3.0-or-later"
+license-files = ["COPYING"]
+requires-python = ">=3.11"
+authors = [
+    { name = "Benjamin Mako Hill" },
+    { name = "Nathan TeBlunthuis" },
+    { name = "Will Beason" },
+    { name = "Sohyeon Hwang" },
+    { name = "Kaylea Champion" },
+]
+keywords = ["mediawiki", "wikipedia", "wiki", "dump", "xml", "revision", "research"]
+classifiers = [
+    "Development Status :: 5 - Production/Stable",
+    "Environment :: Console",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)",
+    "Operating System :: POSIX",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Scientific/Engineering :: Information Analysis",
+    "Topic :: Text Processing :: Markup",
+]
 dependencies = [
     "deltas>=0.7.0",
-    "mediawiki-utilities>=0.4.18",
     "more-itertools>=10.7.0",
     "mwparserfromhell>=0.6.0",
     "mwpersistence>=0.2.4",
     "mwreverts>=0.1.5",
     "mwtypes>=0.4.0",
-    "mwxml>=0.3.6",
+    "mwxml>=0.3.8",
     "pyarrow>=20.0.0",
-    "pyspark>=3.5.0",
-    "pywikidiff2",
     "sortedcontainers>=2.4.0",
-    "yamlconf>=0.2.6",
 ]
 
+# Two features carry dependencies that are awkward enough to install that they
+# are kept out of the base install: -p legacy needs mediawiki-utilities, a 2015
+# package that also pulls in a MySQL client, and --diff and -p wikidiff2 need
+# pywikidiff2, which compiles a C++ extension. Everything else, including the
+# default -p sequence, works with neither.
+[project.optional-dependencies]
+legacy = ["mediawiki-utilities>=0.4.18"]
+# A wikidiff2 extra belongs here too, but pywikidiff2 is not yet published to
+# PyPI, so an extra naming it could not resolve. Until it is, wikiq reports the
+# git install command when --diff or -p wikidiff2 is used without it.
+
+# Duplicated from [dependency-groups] below, which is PEP 735 and so is visible
+# only to uv. This is what makes `pip install -e '.[dev]'` work.
+dev = [
+    "pandas>=2.1.0",
+    "pytest>=8.4.1",
+    "pytest-asyncio>=1.0.0",
+    "pytest-benchmark>=5.1.0",
+]
+
+[project.urls]
+Homepage = "https://wiki.communitydata.science/"
+Repository = "https://gitea.communitydata.science/collective/wikiq"
+
 [project.scripts]
 wikiq = "wikiq:main"
-wikiq-spark = "wikiq_spark:main"
 
 [build-system]
 requires = ["hatchling"]
 build-backend = "hatchling.build"
 
 [tool.hatch.build.targets.wheel]
-packages = ["src/wikiq", "src/wikiq_spark"]
+packages = ["src/wikiq"]
 
-[tool.uv.sources]
-
-yamlconf = { git = "https://github.com/groceryheist/yamlconf" }
-mwxml = { git = "https://github.com/groceryheist/python-mwxml" }
-deltas = { git = "https://github.com/groceryheist/deltas" }
-pywikidiff2 = { git = "ssh://gitea@gitea.communitydata.science:2200/groceryheist/pywikidiff2.git"}
+# Without an explicit list hatchling walks the whole working tree, which sweeps
+# in any local virtualenv and editor state while still omitting the test dumps,
+# since those are gitignored. Tests run from a clone of the repository rather
+# than from the sdist.
+[tool.hatch.build.targets.sdist]
+include = [
+    "src/wikiq",
+    "README.md",
+    "CHANGELOG.md",
+    "COPYING",
+    "pyproject.toml",
+]
 
 [dependency-groups]
 dev = [

runtest.sh

@@ -1,2 +1,6 @@
 #!/usr/bin/env bash
-uv run pytest test/test_wiki_diff_matcher.py --capture=tee-sys
+# Run the test suite from the repository root, which some tests require
+# because they open fixture files by relative path.
+set -euo pipefail
+cd "$(dirname "$0")"
+exec python -m pytest test/ "$@"

src/wikiq/__init__.py

@@ -1,5 +1,21 @@
 #!/usr/bin/env python3
 
+# Copyright (C) 2015-2026 Benjamin Mako Hill, Nathan TeBlunthuis, Will
+# Beason, Sohyeon Hwang, Kaylea Champion, and other contributors
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation, either version 3 of the License, or (at your
+# option) any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program. If not, see <https://www.gnu.org/licenses/>.
+
 # original wikiq headers are: title articleid revid date_time anon
 # editor editor_id minor text_size text_entropy text_md5 reversion
 # additions_size deletions_size
@@ -16,38 +32,78 @@ from collections import deque, defaultdict
 from hashlib import sha1
 from io import TextIOWrapper
 from itertools import groupby
+from pathlib import Path
 from subprocess import PIPE, Popen
 from typing import IO, Any, Generator, TextIO, Union
 
 import mwpersistence
 import mwreverts
 import mwxml
-import pywikidiff2
+import pyarrow as pa
+import pyarrow.csv as pacsv
+from deltas import SegmentMatcher, SequenceMatcher
 from deltas.tokenizers import wikitext_split
 from more_itertools import peekable
 from mwxml import Dump
+
 import wikiq.tables as tables
+from wikiq.resume import get_resume_point
 from wikiq.tables import RevisionTable
-from wikiq.wiki_diff_matcher import WikiDiffMatcher
 from wikiq.wikitext_parser import WikitextParser
-from wikiq.resume import (
-    get_checkpoint_path,
-    read_checkpoint,
-    get_resume_point,
-    setup_resume_temp_output,
-    finalize_resume_merge,
-    cleanup_interrupted_resume,
-)
+
+try:
+    from importlib.metadata import PackageNotFoundError
+    from importlib.metadata import version as _package_version
+    __version__ = _package_version("wikiq")
+except PackageNotFoundError:
+    __version__ = "unknown"
 
 TO_ENCODE = ("title", "editor")
 PERSISTENCE_RADIUS = 7
 DIFF_TIMEOUT_MS = 60000
-from pathlib import Path
 
-import pyarrow as pa
-import pyarrow.csv as pacsv
-import pyarrow.parquet as pq
-from deltas import SegmentMatcher, SequenceMatcher
+
+# Some dependencies serve a single feature and are awkward enough to install
+# that wikiq leaves them out of the base install. Each is imported inside the
+# code path that needs it, so importing wikiq works without them and a plain
+# `pip install` needs no C++ compiler. These helpers do that import and, when
+# it fails, explain what to install rather than raising an ImportError.
+
+def require_pywikidiff2(feature: str):
+    """Import and return pywikidiff2, or exit explaining how to install it.
+
+    feature names the wikiq option that needs it, so the message points at
+    whatever the user actually asked for.
+    """
+    try:
+        import pywikidiff2
+    except ImportError:
+        raise SystemExit(
+            f"{feature} requires pywikidiff2, which wikiq does not install by "
+            "default because it compiles a C++ extension.\n"
+            "Install it with:\n"
+            "    pip install 'pywikidiff2 @ git+"
+            "https://gitea.communitydata.science/groceryheist/pywikidiff2.git'\n"
+            "A C++ compiler and libthai must be available. Other persistence "
+            "methods (-p sequence, -p segment, -p legacy) do not need it."
+        )
+    return pywikidiff2
+
+
+def require_mw_persistence():
+    """Import and return mw.lib.persistence, or exit explaining how to get it."""
+    try:
+        from mw.lib import persistence
+    except ImportError:
+        raise SystemExit(
+            "-p legacy requires mediawiki-utilities, which wikiq no longer "
+            "installs by default.\n"
+            "Install it with:\n"
+            "    pip install 'wikiq[legacy]'\n"
+            "This method exists to reproduce results from older research "
+            "projects; -p sequence is the current equivalent."
+        )
+    return persistence
 
 
 def pyarrow_type_to_spark(pa_type):
@@ -118,14 +174,19 @@ def build_table(
     wikilinks: bool = False,
     templates: bool = False,
     headings: bool = False,
+    content_sizes: bool = False,
+    redirect_aliases: Union[list[str], None] = None,
 ):
     """Build the RevisionTable with appropriate columns based on flags.
 
     Returns:
-        (table, reverts_column) - the table and a reference to the reverts column
-        (which process() needs for setting the revert detector).
+        (table, reverts_column, wikitext_parser) - the table, a reference to
+        the reverts column (which process() needs for setting the revert
+        detector), and the wikitext parser (None unless a wikitext-parsing
+        column was requested).
     """
     reverts_column = tables.RevisionReverts()
+    redirect_detector = tables.RedirectDetector(redirect_aliases)
 
     table = RevisionTable([
         tables.RevisionId(),
@@ -133,7 +194,8 @@ def build_table(
         tables.RevisionArticleId(),
         tables.RevisionPageTitle(),
         tables.RevisionNamespace(),
-        tables.RevisionRedirectTarget(),
+        tables.RevisionIsRedirect(redirect_detector),
+        tables.RevisionRedirectTarget(redirect_detector),
         tables.RevisionDeleted(),
         tables.RevisionEditorId(),
         tables.RevisionEditSummary(),
@@ -152,7 +214,7 @@ def build_table(
         table.columns.append(tables.RevisionCollapsed())
 
     wikitext_parser = None
-    if external_links or citations or wikilinks or templates or headings:
+    if external_links or citations or wikilinks or templates or headings or content_sizes:
         wikitext_parser = WikitextParser()
 
         if external_links:
@@ -170,6 +232,10 @@ def build_table(
         if headings:
             table.columns.append(tables.RevisionHeadings(wikitext_parser))
 
+        if content_sizes:
+            table.columns.append(tables.RevisionContentChars(wikitext_parser))
+            table.columns.append(tables.RevisionMarkupChars(wikitext_parser))
+
         table.columns.append(tables.RevisionParserTimeout(wikitext_parser))
 
     return table, reverts_column, wikitext_parser
@@ -391,8 +457,12 @@ class WikiqPage:
         # page.namespace is inconsistent with namespace_map
         if page.namespace not in namespace_map:
             page.namespace = 0
+        # mwxml >= 0.3.7 keeps the namespace prefix on the title when the dump
+        # has <ns> tags, but strips it on the no-<ns> fallback path
         if page.namespace != 0:
-            page.title = ":".join([namespace_map[page.namespace], page.title])
+            ns_prefix = namespace_map[page.namespace] + ":"
+            if not page.title.startswith(ns_prefix):
+                page.title = ns_prefix + page.title
         self.restrictions = page.restrictions
         self.collapse_user = collapse_user
         self.mwpage = page
@@ -463,7 +533,8 @@ class RegexPair(object):
         # if there are named capture groups in the regex
         if self.has_groups:
             # if there are matches of some sort in this revision content, fill the lists for each cap_group
-            if self.pattern.search(content) is not None:
+            # content can be None when the text or comment was deleted/suppressed
+            if content is not None and self.pattern.search(content) is not None:
                 m = self.pattern.finditer(content)
                 matchobjects = list(m)
 
@@ -491,12 +562,11 @@ class RegexPair(object):
         # there are no capture groups, we just search for all the matches of the regex
         else:
             # given that there are matches to be made
-            if type(content) in (str, bytes):
-                if self.pattern.search(content) is not None:
-                    m = self.pattern.findall(content)
-                    temp_dict[self.label] = ", ".join(m)
-                else:
-                    temp_dict[self.label] = None
+            if content is not None and self.pattern.search(content) is not None:
+                m = self.pattern.findall(content)
+                temp_dict[self.label] = ", ".join(m)
+            else:
+                temp_dict[self.label] = None
 
         return temp_dict
 
@@ -518,26 +588,23 @@ class WikiqParser:
         revert_radius: int = 15,
         output_jsonl: bool = False,
         output_jsonl_dir: bool = False,
-        output_parquet: bool = False,
         batch_size: int = 1024,
-        resume_point: Union[tuple, dict, None] = None,
-        partition_namespaces: bool = False,
+        resume_point: Union[tuple, None] = None,
         external_links: bool = False,
         citations: bool = False,
         wikilinks: bool = False,
         templates: bool = False,
         headings: bool = False,
+        content_sizes: bool = False,
+        redirect_aliases: Union[list[str], None] = None,
         time_limit_seconds: Union[float, None] = None,
-        max_revisions_per_file: int = 0,
         input_filename: Union[str, None] = None,
     ):
         """
         Parameters:
            persist : what persistence method to use. Takes a PersistMethod value
-           resume_point : if set, either a (pageid, revid) tuple for single-file output,
-                          or a dict mapping namespace -> (pageid, revid) for partitioned output.
-                          For single-file: skip all revisions up to and including this point.
-           max_revisions_per_file : if > 0, close and rotate output files after this many revisions
+           resume_point : if set, a (pageid, revid) tuple; skip all revisions up
+                          to and including this point.
            input_filename : original input filename (needed for .jsonl.d output to derive output filename)
         """
         self.input_file = input_file
@@ -549,16 +616,16 @@ class WikiqParser:
         self.revert_radius = revert_radius
         self.diff = diff
         self.text = text
-        self.partition_namespaces = partition_namespaces
         self.resume_point = resume_point
         self.external_links = external_links
         self.citations = citations
         self.wikilinks = wikilinks
         self.templates = templates
         self.headings = headings
+        self.content_sizes = content_sizes
+        self.redirect_aliases = redirect_aliases
         self.shutdown_requested = False
         self.time_limit_seconds = time_limit_seconds
-        self.max_revisions_per_file = max_revisions_per_file
         if namespaces is not None:
             self.namespace_filter = set(namespaces)
         else:
@@ -576,13 +643,9 @@ class WikiqParser:
         self.batch_size = batch_size
         self.output_jsonl = output_jsonl
         self.output_jsonl_dir = output_jsonl_dir
-        self.output_parquet = output_parquet
         self.output_file = output_file
 
-        if output_parquet:
-            self.pq_writer = None
-            self.parquet_buffer = []
-        elif output_jsonl:
+        if output_jsonl:
             pass  # JSONLWriter created in process()
         else:
             # TSV output
@@ -592,10 +655,6 @@ class WikiqParser:
             else:
                 self.output_file = open(output_file, "wb")
 
-        # Checkpoint for tracking resume point (path only, no open file handle for NFS safety)
-        self.checkpoint_path = None
-        self.checkpoint_state = {}  # namespace -> (pageid, revid) or None -> (pageid, revid)
-
     def request_shutdown(self):
         """Request graceful shutdown. The process() method will exit after completing the current batch."""
         self.shutdown_requested = True
@@ -620,76 +679,6 @@ class WikiqParser:
         if timer is not None:
             timer.cancel()
 
-    def _get_part_path(self, base_path, part_num):
-        """Generate path with part number inserted before extension.
-
-        Example: output.parquet -> output.part0.parquet
-        """
-        path = Path(base_path)
-        return path.parent / f"{path.stem}.part{part_num}{path.suffix}"
-
-    def _open_checkpoint(self, output_file):
-        """Enable checkpointing for Parquet output only.
-
-        JSONL doesn't need checkpoint files - resume point is derived from last line.
-        """
-        if not self.output_parquet or output_file == sys.stdout.buffer:
-            return
-        self.checkpoint_path = get_checkpoint_path(output_file, self.partition_namespaces)
-        Path(self.checkpoint_path).parent.mkdir(parents=True, exist_ok=True)
-        print(f"Checkpoint enabled: {self.checkpoint_path}", file=sys.stderr)
-
-    def _update_checkpoint(self, pageid, revid, namespace=None, part=0):
-        """Update checkpoint state and write atomically (NFS-safe)."""
-        if self.checkpoint_path is None:
-            return
-        if self.partition_namespaces:
-            self.checkpoint_state[namespace] = {"pageid": pageid, "revid": revid, "part": part}
-        else:
-            self.checkpoint_state = {"pageid": pageid, "revid": revid, "part": part}
-        # Atomic write: write to temp file, then rename
-        temp_path = self.checkpoint_path + ".tmp"
-        with open(temp_path, 'w') as f:
-            json.dump(self.checkpoint_state, f)
-        os.replace(temp_path, self.checkpoint_path)
-
-    def _close_checkpoint(self, delete=False):
-        """Clean up checkpoint, optionally deleting it."""
-        if self.checkpoint_path is None:
-            return
-        if delete and os.path.exists(self.checkpoint_path):
-            os.remove(self.checkpoint_path)
-            print(f"Checkpoint deleted: {self.checkpoint_path}", file=sys.stderr)
-        elif os.path.exists(self.checkpoint_path):
-            print(f"Checkpoint preserved for resume: {self.checkpoint_path}", file=sys.stderr)
-        # Clean up any leftover temp file
-        temp_path = self.checkpoint_path + ".tmp"
-        if os.path.exists(temp_path):
-            os.remove(temp_path)
-
-    def _write_batch(self, row_buffer, schema, writer, pq_writers, ns_base_paths, sorting_cols, namespace=None, part_numbers=None):
-        """Write a batch of rows to the appropriate writer.
-
-        For partitioned output, creates writer lazily if needed.
-        Returns (writer, num_rows) - writer used and number of rows written.
-        """
-        num_rows = len(row_buffer.get("revid", []))
-        if self.partition_namespaces and namespace is not None:
-            if namespace not in pq_writers:
-                base_path = ns_base_paths[namespace]
-                part_num = part_numbers.get(namespace, 0) if part_numbers else 0
-                if self.max_revisions_per_file > 0:
-                    ns_path = self._get_part_path(base_path, part_num)
-                else:
-                    ns_path = base_path
-                Path(ns_path).parent.mkdir(exist_ok=True, parents=True)
-                pq_writers[namespace] = pq.ParquetWriter(
-                    ns_path, schema, flavor="spark", sorting_columns=sorting_cols
-                )
-            writer = pq_writers[namespace]
-        writer.write(pa.record_batch(row_buffer, schema=schema))
-        return writer, num_rows
-
     def make_matchmake_pairs(self, patterns, labels) -> list[RegexPair]:
         if (patterns is not None and labels is not None) and (
             len(patterns) == len(labels)
@@ -747,26 +736,7 @@ class WikiqParser:
         time_limit_timer = self._start_time_limit_timer()
 
         # Track whether we've passed the resume point
-        if self.resume_point is None:
-            found_resume_point = True
-        elif self.partition_namespaces:
-            found_resume_point = {}
-        else:
-            found_resume_point = False
-
-        # When resuming with parquet, write new data to temp file/directory and merge at the end
-        original_output_file = None
-        temp_output_file = None
-        original_partition_dir = None
-        if self.resume_point is not None and self.output_parquet:
-            original_output_file, temp_output_file, original_partition_dir = \
-                setup_resume_temp_output(self.output_file, self.partition_namespaces)
-            if temp_output_file is not None:
-                self.output_file = temp_output_file
-
-        # Open checkpoint file for tracking resume point
-        checkpoint_output = original_output_file if original_output_file else self.output_file
-        self._open_checkpoint(checkpoint_output)
+        found_resume_point = self.resume_point is None
 
         # Construct dump file iterator
         dump = WikiqIterator(self.input_file, collapse_user=self.collapse_user)
@@ -779,6 +749,8 @@ class WikiqParser:
             wikilinks=self.wikilinks,
             templates=self.templates,
             headings=self.headings,
+            content_sizes=self.content_sizes,
+            redirect_aliases=self.redirect_aliases,
         )
 
         # Extract list of namespaces
@@ -804,48 +776,8 @@ class WikiqParser:
 
         # Initialize writer
         writer = None
-        sorting_cols = None
-        ns_base_paths = {}
-        pq_writers = {}
-        part_numbers = {}
 
-        if self.output_parquet:
-            pageid_sortingcol = pq.SortingColumn(schema.get_field_index("articleid"))
-            revid_sortingcol = pq.SortingColumn(schema.get_field_index("revid"))
-            sorting_cols = [pageid_sortingcol, revid_sortingcol]
-
-            if self.resume_point is not None:
-                if self.partition_namespaces:
-                    for ns, resume_data in self.resume_point.items():
-                        part_numbers[ns] = resume_data[2] if len(resume_data) > 2 else 0
-                else:
-                    part_numbers[None] = self.resume_point[2] if len(self.resume_point) > 2 else 0
-
-            if not self.partition_namespaces:
-                if self.max_revisions_per_file > 0:
-                    output_path_with_part = self._get_part_path(self.output_file, part_numbers.get(None, 0))
-                else:
-                    output_path_with_part = self.output_file
-                writer = pq.ParquetWriter(
-                    output_path_with_part,
-                    schema,
-                    flavor="spark",
-                    sorting_columns=sorting_cols,
-                )
-            else:
-                output_path = Path(self.output_file)
-                if self.namespace_filter is not None:
-                    namespaces = self.namespace_filter
-                else:
-                    namespaces = self.namespaces.values()
-                ns_base_paths = {
-                    ns: (output_path.parent / f"namespace={ns}") / output_path.name
-                    for ns in namespaces
-                }
-                for ns in namespaces:
-                    if ns not in part_numbers:
-                        part_numbers[ns] = 0
-        elif self.output_jsonl:
+        if self.output_jsonl:
             append_mode = self.resume_point is not None
             if self.output_jsonl_dir:
                 # Create directory for JSONL output
@@ -871,6 +803,7 @@ class WikiqParser:
         differ = None
         fast_differ = None
         if self.diff:
+            pywikidiff2 = require_pywikidiff2("--diff")
             differ = pywikidiff2.pywikidiff2(
                 num_context_lines=1000000,
                 max_word_level_diff_complexity=-1,
@@ -891,34 +824,16 @@ class WikiqParser:
         # Write buffer: accumulate rows before flushing
         write_buffer = defaultdict(list)
         buffer_count = 0
-        last_namespace = None
 
         def flush_buffer():
-            nonlocal write_buffer, buffer_count, last_namespace
+            nonlocal write_buffer, buffer_count
             if buffer_count == 0:
                 return
             row_buffer = dict(write_buffer)
-            namespace = last_namespace
-            if self.output_parquet:
-                if self.partition_namespaces:
-                    self._write_batch(
-                        row_buffer, schema, writer, pq_writers, ns_base_paths,
-                        sorting_cols, namespace=namespace, part_numbers=part_numbers
-                    )
-                else:
-                    writer.write(pa.record_batch(row_buffer, schema=schema))
-            elif self.output_jsonl:
+            if self.output_jsonl:
                 writer.write_batch(row_buffer)
             else:
                 writer.write(pa.record_batch(row_buffer, schema=schema))
-
-            # Update checkpoint
-            last_pageid = row_buffer["articleid"][-1]
-            last_revid = row_buffer["revid"][-1]
-            part = part_numbers.get(namespace if self.partition_namespaces else None, 0)
-            self._update_checkpoint(last_pageid, last_revid,
-                                    namespace=namespace if self.partition_namespaces else None,
-                                    part=part)
             write_buffer = defaultdict(list)
             buffer_count = 0
 
@@ -976,12 +891,14 @@ class WikiqParser:
                         revert_radius=PERSISTENCE_RADIUS,
                     )
                 elif self.persist == PersistMethod.wikidiff2:
+                    require_pywikidiff2("-p wikidiff2")
+                    from wikiq.wiki_diff_matcher import WikiDiffMatcher
                     wikidiff_matcher = WikiDiffMatcher(tokenizer=wikitext_split)
                     persist_state = mwpersistence.DiffState(
                         wikidiff_matcher, revert_radius=PERSISTENCE_RADIUS
                     )
                 else:
-                    from mw.lib import persistence
+                    persistence = require_mw_persistence()
                     persist_state = persistence.State()
 
             # Pending persistence values waiting for window to fill
@@ -1103,7 +1020,6 @@ class WikiqParser:
                         for k, v in oldest_row.items():
                             write_buffer[k].append(v)
                         buffer_count += 1
-                        last_namespace = page.mwpage.namespace
 
                         if buffer_count >= self.batch_size:
                             flush_buffer()
@@ -1120,7 +1036,6 @@ class WikiqParser:
                     for k, v in row.items():
                         write_buffer[k].append(v)
                     buffer_count += 1
-                    last_namespace = page.mwpage.namespace
 
                     if buffer_count >= self.batch_size:
                         flush_buffer()
@@ -1146,7 +1061,6 @@ class WikiqParser:
                     for k, v in pending_row.items():
                         write_buffer[k].append(v)
                     buffer_count += 1
-                    last_namespace = page.mwpage.namespace
 
             if self.shutdown_requested:
                 break
@@ -1170,24 +1084,9 @@ class WikiqParser:
         )
 
         # Close all writers
-        if self.output_parquet and self.partition_namespaces:
-            for pq_writer in pq_writers.values():
-                pq_writer.close()
-        elif writer is not None:
+        if writer is not None:
             writer.close()
 
-        # Close checkpoint file; delete it only if we completed without interruption
-        self._close_checkpoint(delete=not self.shutdown_requested)
-
-        # Merge temp output with original for parquet resume
-        if original_output_file is not None and temp_output_file is not None:
-            finalize_resume_merge(
-                original_output_file,
-                temp_output_file,
-                self.partition_namespaces,
-                original_partition_dir
-            )
-
 def match_archive_suffix(input_filename):
     if re.match(r".*\.7z$", input_filename):
         cmd = ["7za", "x", "-so", input_filename]
@@ -1215,14 +1114,12 @@ def get_output_filename(input_filename, output_format='tsv') -> str:
 
     Args:
         input_filename: Input dump file path
-        output_format: 'tsv', 'jsonl', or 'parquet'
+        output_format: 'tsv' or 'jsonl'
     """
     output_filename = re.sub(r"\.(7z|gz|bz2)?$", "", input_filename)
     output_filename = re.sub(r"\.xml", "", output_filename)
     if output_format == 'jsonl':
         output_filename = output_filename + ".jsonl"
-    elif output_format == 'parquet':
-        output_filename = output_filename + ".parquet"
     else:
         output_filename = output_filename + ".tsv"
     return output_filename
@@ -1249,7 +1146,13 @@ def main():
         dest="output",
         type=str,
         nargs=1,
-        help="Output file or directory. Format is detected from extension: .jsonl for JSONL, .parquet for Parquet, otherwise TSV.",
+        help="Output file or directory. Format is detected from extension: .jsonl for JSONL, otherwise TSV.",
+    )
+
+    parser.add_argument(
+        "--version",
+        action="version",
+        version=f"wikiq {__version__}",
     )
 
     parser.add_argument(
@@ -1403,6 +1306,23 @@ def main():
         help="Extract section headings from each revision.",
     )
 
+    parser.add_argument(
+        "--content-sizes",
+        dest="content_sizes",
+        action="store_true",
+        default=False,
+        help="Split each revision into content_chars, the information it carries, and markup_chars, the wikitext formatting around it. Content is what someone who did not know wikitext would have typed: words, link labels, and template parameter values. The two columns sum to the revision length.",
+    )
+
+    parser.add_argument(
+        "--redirect-aliases",
+        dest="redirect_aliases",
+        default=None,
+        type=str,
+        action="append",
+        help="Additional localized redirect keyword recognized alongside #REDIRECT when detecting redirect revisions (e.g. OMDIRIGERING on Swedish wikis). May be given multiple times or as a comma-separated list.",
+    )
+
     parser.add_argument(
         "--fandom-2020",
         dest="fandom_2020",
@@ -1415,7 +1335,7 @@ def main():
         dest="batch_size",
         default=1500,
         type=int,
-        help="How many revisions to process in each batch. This ends up being the Parquet row group size",
+        help="How many revisions to process in each batch.",
     )
 
     parser.add_argument(
@@ -1433,22 +1353,6 @@ def main():
         help="Time limit in hours before graceful shutdown. Set to 0 to disable (default).",
     )
 
-    parser.add_argument(
-        "--partition-namespaces",
-        dest="partition_namespaces",
-        action="store_true",
-        default=False,
-        help="For Parquet output, partition output by namespace into separate files.",
-    )
-
-    parser.add_argument(
-        "--max-revisions-per-file",
-        dest="max_revisions_per_file",
-        type=int,
-        default=0,
-        help="For Parquet output, split output into multiple files after this many revisions. Set to 0 to disable (default).",
-    )
-
     args = parser.parse_args()
 
     # set persistence method
@@ -1464,11 +1368,30 @@ def main():
     else:
         persist = PersistMethod.sequence
 
+    # Check for the optional dependencies up front. Both are used deep in the
+    # per-revision loop, and a run can stream for hours before reaching them.
+    if args.diff:
+        require_pywikidiff2("--diff")
+    if persist == PersistMethod.wikidiff2:
+        require_pywikidiff2("-p wikidiff2")
+    elif persist == PersistMethod.legacy:
+        require_mw_persistence()
+
     if args.namespace_filter is not None:
         namespaces = args.namespace_filter
     else:
         namespaces = None
 
+    # --redirect-aliases may be given multiple times or comma-separated
+    redirect_aliases = None
+    if args.redirect_aliases:
+        redirect_aliases = [
+            alias.strip()
+            for arg in args.redirect_aliases
+            for alias in arg.split(",")
+            if alias.strip()
+        ]
+
     # Handle --print-schema: build and output schema, then exit
     if args.print_schema:
         regex_revision_pairs = make_regex_pairs(args.regex_match_revision, args.regex_revision_label)
@@ -1482,6 +1405,8 @@ def main():
             wikilinks=args.wikilinks,
             templates=args.templates,
             headings=args.headings,
+            content_sizes=args.content_sizes,
+            redirect_aliases=redirect_aliases,
         )
         schema = build_schema(
             table,
@@ -1496,6 +1421,7 @@ def main():
         print(json.dumps(spark_schema, indent=2))
         sys.exit(0)
 
+    print(f"wikiq {__version__}", file=sys.stderr)
     print(args, file=sys.stderr)
     if len(args.dumpfiles) > 0:
         for filename in args.dumpfiles:
@@ -1508,13 +1434,11 @@ def main():
             # Detect output format from extension
             output_jsonl_dir = output.endswith(".jsonl.d")
             output_jsonl = output.endswith(".jsonl") or output_jsonl_dir
-            output_parquet = output.endswith(".parquet")
-            partition_namespaces = args.partition_namespaces and output_parquet
 
             if args.stdout:
                 output_file = sys.stdout.buffer
-            elif output_jsonl or output_parquet:
-                # Output is a JSONL or Parquet file path - use it directly
+            elif output_jsonl:
+                # Output is a JSONL file path - use it directly
                 output_file = output
             elif os.path.isdir(output):
                 # Output is a directory - derive filename from input
@@ -1526,25 +1450,14 @@ def main():
             # Handle resume functionality before opening input file
             resume_point = None
             if args.resume:
-                if (output_jsonl or output_parquet) and not args.stdout:
-                    # Clean up any interrupted resume from previous run
-                    if output_parquet:
-                        cleanup_result = cleanup_interrupted_resume(output_file, partition_namespaces)
-                        if cleanup_result == "start_fresh":
-                            resume_point = None
-                        else:
-                            resume_point = get_resume_point(output_file, partition_namespaces)
-                    else:
-                        # JSONL: get resume point from last line of file (no checkpoint)
-                        resume_point = get_resume_point(output_file, input_file=filename)
+                if output_jsonl and not args.stdout:
+                    # JSONL: get resume point from last line of file
+                    resume_point = get_resume_point(output_file, input_file=filename)
                     if resume_point is not None:
-                        if isinstance(resume_point, dict):
-                            print(f"Resuming from checkpoint for {len(resume_point)} namespaces", file=sys.stderr)
-                        else:
-                            pageid, revid = resume_point[0], resume_point[1]
-                            print(f"Resuming from checkpoint: pageid={pageid}, revid={revid}", file=sys.stderr)
+                        pageid, revid = resume_point[0], resume_point[1]
+                        print(f"Resuming from: pageid={pageid}, revid={revid}", file=sys.stderr)
                 else:
-                    sys.exit("Error: --resume only works with JSONL or Parquet output (not stdout or TSV)")
+                    sys.exit("Error: --resume only works with JSONL output (not stdout or TSV)")
 
             # Now open the input file
             print("Processing file: %s" % filename, file=sys.stderr)
@@ -1567,8 +1480,6 @@ def main():
                 diff=args.diff,
                 output_jsonl=output_jsonl,
                 output_jsonl_dir=output_jsonl_dir,
-                output_parquet=output_parquet,
-                partition_namespaces=partition_namespaces,
                 batch_size=args.batch_size,
                 resume_point=resume_point,
                 external_links=args.external_links,
@@ -1576,8 +1487,9 @@ def main():
                 wikilinks=args.wikilinks,
                 templates=args.templates,
                 headings=args.headings,
+                content_sizes=args.content_sizes,
+                redirect_aliases=redirect_aliases,
                 time_limit_seconds=time_limit_seconds,
-                max_revisions_per_file=args.max_revisions_per_file,
                 input_filename=filename,
             )
 
@@ -1631,6 +1543,8 @@ def main():
             wikilinks=args.wikilinks,
             templates=args.templates,
             headings=args.headings,
+            content_sizes=args.content_sizes,
+            redirect_aliases=redirect_aliases,
             time_limit_seconds=time_limit_seconds,
         )
 

src/wikiq/diff_pyarrow_schema.py

@@ -1,3 +1,19 @@
+# Copyright (C) 2015-2026 Benjamin Mako Hill, Nathan TeBlunthuis, Will
+# Beason, Sohyeon Hwang, Kaylea Champion, and other contributors
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation, either version 3 of the License, or (at your
+# option) any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program. If not, see <https://www.gnu.org/licenses/>.
+
 import pyarrow as pa
 
 # Schema for the `highlightRanges` object, an array of which can be nested in a diff object.

src/wikiq/resume.py

@@ -1,10 +1,24 @@
-"""
-Checkpoint and resume functionality for wikiq output.
+# Copyright (C) 2015-2026 Benjamin Mako Hill, Nathan TeBlunthuis, Will
+# Beason, Sohyeon Hwang, Kaylea Champion, and other contributors
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation, either version 3 of the License, or (at your
+# option) any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program. If not, see <https://www.gnu.org/licenses/>.
 
-This module handles:
-- Finding resume points in existing output (JSONL or Parquet)
-- Merging resumed data with existing output (for Parquet, streaming, memory-efficient)
-- Checkpoint file management for fast resume point lookup
+"""
+Resume functionality for wikiq JSONL output.
+
+The resume point is derived from the last complete line of the existing
+output file, so no separate checkpoint files are needed.
 """
 
 import json
@@ -12,147 +26,6 @@ import os
 import sys
 from collections import deque
 
-import pyarrow.parquet as pq
-
-
-def get_checkpoint_path(output_file, partition_namespaces=False):
-    """Get the path to the checkpoint file for a given output file.
-
-    For partitioned output, the checkpoint is placed outside the partition directory
-    to avoid pyarrow trying to read it as a parquet file. The filename includes
-    the output filename to keep it unique per input file (for parallel jobs).
-    """
-    if partition_namespaces:
-        partition_dir = os.path.dirname(output_file)
-        output_filename = os.path.basename(output_file)
-        parent_dir = os.path.dirname(partition_dir)
-        return os.path.join(parent_dir, output_filename + ".checkpoint")
-    return str(output_file) + ".checkpoint"
-
-
-def read_checkpoint(checkpoint_path, partition_namespaces=False):
-    """
-    Read resume point from checkpoint file if it exists.
-
-    Checkpoint format:
-        Single file: {"pageid": 54, "revid": 325} or {"pageid": 54, "revid": 325, "part": 2}
-        Partitioned: {"0": {"pageid": 54, "revid": 325, "part": 1}, ...}
-
-    Returns:
-        For single files: A tuple (pageid, revid) or (pageid, revid, part), or None if not found.
-        For partitioned: A dict mapping namespace -> (pageid, revid, part), or None.
-    """
-    if not os.path.exists(checkpoint_path):
-        return None
-
-    try:
-        with open(checkpoint_path, 'r') as f:
-            data = json.load(f)
-
-        if not data:
-            return None
-
-        # Single-file format: {"pageid": ..., "revid": ..., "part": ...}
-        if "pageid" in data and "revid" in data:
-            part = data.get("part", 0)
-            if part > 0:
-                return (data["pageid"], data["revid"], part)
-            return (data["pageid"], data["revid"])
-
-        # Partitioned format: {"0": {"pageid": ..., "revid": ..., "part": ...}, ...}
-        result = {}
-        for key, value in data.items():
-            part = value.get("part", 0)
-            result[int(key)] = (value["pageid"], value["revid"], part)
-
-        return result if result else None
-
-    except (json.JSONDecodeError, IOError, KeyError, TypeError) as e:
-        print(f"Warning: Could not read checkpoint file {checkpoint_path}: {e}", file=sys.stderr)
-        return None
-
-
-def cleanup_interrupted_resume(output_file, partition_namespaces):
-    """
-    Merge any leftover .resume_temp files from a previous interrupted run.
-
-    This should be called BEFORE get_resume_point() so the resume point
-    is calculated from the merged data.
-
-    Returns:
-        None - no temp files found or normal merge completed
-        "start_fresh" - both original and temp were corrupted and deleted
-    """
-    import shutil
-
-    if partition_namespaces:
-        partition_dir = os.path.dirname(output_file)
-        output_filename = os.path.basename(output_file)
-        temp_suffix = ".resume_temp"
-
-        if not os.path.isdir(partition_dir):
-            return
-
-        has_old_temp_files = False
-        for ns_dir in os.listdir(partition_dir):
-            if ns_dir.startswith('namespace='):
-                temp_path = os.path.join(partition_dir, ns_dir, output_filename + temp_suffix)
-                if os.path.exists(temp_path):
-                    has_old_temp_files = True
-                    break
-
-        if has_old_temp_files:
-            print(f"Found leftover temp files in {partition_dir} from previous interrupted partitioned run, merging first...", file=sys.stderr)
-            had_corruption = merge_partitioned_namespaces(partition_dir, temp_suffix, output_filename)
-
-            has_valid_data = False
-            for ns_dir in os.listdir(partition_dir):
-                if ns_dir.startswith('namespace='):
-                    ns_path = os.path.join(partition_dir, ns_dir)
-                    parquet_files = [f for f in os.listdir(ns_path) if f.endswith('.parquet') and not f.endswith('.resume_temp')]
-                    if parquet_files:
-                        has_valid_data = True
-                        break
-
-            if had_corruption and not has_valid_data:
-                checkpoint_path = get_checkpoint_path(output_file, partition_namespaces)
-                if os.path.exists(checkpoint_path):
-                    os.remove(checkpoint_path)
-                print("All partitioned files were corrupted, will start fresh.", file=sys.stderr)
-                return "start_fresh"
-
-            print("Previous temp files merged successfully.", file=sys.stderr)
-    else:
-        temp_output_file = output_file + ".resume_temp"
-        if os.path.exists(temp_output_file) and not os.path.isdir(temp_output_file):
-            print(f"Found leftover temp file {temp_output_file} from previous interrupted run, merging first...", file=sys.stderr)
-            merged_path = output_file + ".merged"
-            merged = merge_parquet_files(output_file, temp_output_file, merged_path)
-            if merged == "original_only":
-                os.remove(temp_output_file)
-            elif merged == "temp_only":
-                if os.path.exists(output_file):
-                    os.remove(output_file)
-                os.rename(temp_output_file, output_file)
-                print("Recovered from temp file (original was corrupted or missing).", file=sys.stderr)
-            elif merged == "both_invalid":
-                if os.path.exists(output_file):
-                    os.remove(output_file)
-                if os.path.exists(temp_output_file):
-                    os.remove(temp_output_file)
-                checkpoint_path = get_checkpoint_path(output_file, partition_namespaces)
-                if os.path.exists(checkpoint_path):
-                    os.remove(checkpoint_path)
-                print("Both files were corrupted, will start fresh.", file=sys.stderr)
-                return "start_fresh"
-            elif merged == "merged":
-                os.remove(output_file)
-                os.rename(merged_path, output_file)
-                os.remove(temp_output_file)
-                print("Previous temp file merged successfully.", file=sys.stderr)
-            else:
-                os.remove(temp_output_file)
-
 
 def get_jsonl_resume_point(output_file, input_file=None):
     """Get resume point from last complete line of JSONL file.
@@ -207,325 +80,18 @@ def get_jsonl_resume_point(output_file, input_file=None):
         return None
 
 
-def get_resume_point(output_file, partition_namespaces=False, input_file=None):
+def get_resume_point(output_file, input_file=None):
     """
-    Find the resume point(s) from existing output.
-
-    For JSONL: reads last line of file (no checkpoint needed).
-    For Parquet: checks checkpoint file, falls back to scanning parquet.
+    Find the resume point from existing JSONL output.
 
     Args:
         output_file: Path to the output file.
-        partition_namespaces: Whether the output uses namespace partitioning.
         input_file: Path to input file (needed for .jsonl.d directory output).
 
     Returns:
-        For single files: A tuple (pageid, revid) or (pageid, revid, part), or None.
-        For partitioned: A dict mapping namespace -> (pageid, revid, part), or None.
+        A (pageid, revid) tuple, or None.
     """
-    # For JSONL, read resume point directly from last line (no checkpoint needed)
-    if output_file.endswith('.jsonl') or output_file.endswith('.jsonl.d'):
-        result = get_jsonl_resume_point(output_file, input_file)
-        if result:
-            print(f"Resume point found from JSONL: pageid={result[0]}, revid={result[1]}", file=sys.stderr)
-        return result
-
-    # For Parquet, use checkpoint file (fast)
-    checkpoint_path = get_checkpoint_path(output_file, partition_namespaces)
-    checkpoint_result = read_checkpoint(checkpoint_path, partition_namespaces)
-    if checkpoint_result is not None:
-        print(f"Resume point found in checkpoint file {checkpoint_path}", file=sys.stderr)
-        return checkpoint_result
-
-    # Fall back to scanning parquet (slow, for backwards compatibility)
-    print(f"No checkpoint file found at {checkpoint_path}, scanning parquet output...", file=sys.stderr)
-    try:
-        if partition_namespaces:
-            return _get_resume_point_partitioned(output_file)
-        else:
-            return _get_resume_point_single_file(output_file)
-    except Exception as e:
-        print(f"Error reading resume point from {output_file}: {e}", file=sys.stderr)
-        return None
-
-
-def _get_last_row_resume_point(pq_path):
-    """Get resume point by reading only the last row group of a parquet file."""
-    pf = pq.ParquetFile(pq_path)
-    if pf.metadata.num_row_groups == 0:
-        return None
-
-    last_rg_idx = pf.metadata.num_row_groups - 1
-    table = pf.read_row_group(last_rg_idx, columns=['articleid', 'revid'])
-    if table.num_rows == 0:
-        return None
-
-    max_pageid = table['articleid'][-1].as_py()
-    max_revid = table['revid'][-1].as_py()
-    return (max_pageid, max_revid, 0)
-
-
-def _get_resume_point_partitioned(output_file):
-    """Find per-namespace resume points from partitioned output."""
-    partition_dir = os.path.dirname(output_file)
-    output_filename = os.path.basename(output_file)
-
-    if not os.path.exists(partition_dir) or not os.path.isdir(partition_dir):
-        return None
-
-    namespace_dirs = [d for d in os.listdir(partition_dir) if d.startswith('namespace=')]
-    if not namespace_dirs:
-        return None
-
-    resume_points = {}
-    for ns_dir in namespace_dirs:
-        ns = int(ns_dir.split('=')[1])
-        pq_path = os.path.join(partition_dir, ns_dir, output_filename)
-
-        if not os.path.exists(pq_path):
-            continue
-
-        try:
-            result = _get_last_row_resume_point(pq_path)
-            if result is not None:
-                resume_points[ns] = result
-        except Exception as e:
-            print(f"Warning: Could not read {pq_path}: {e}", file=sys.stderr)
-            continue
-
-    return resume_points if resume_points else None
-
-
-def _get_resume_point_single_file(output_file):
-    """Find resume point from a single parquet file."""
-    if not os.path.exists(output_file):
-        return None
-
-    if os.path.isdir(output_file):
-        return None
-
-    return _get_last_row_resume_point(output_file)
-
-
-def merge_parquet_files(original_path, temp_path, merged_path):
-    """
-    Merge two parquet files by streaming row groups.
-
-    Returns:
-        "merged" - merged file was created from both sources
-        "original_only" - temp was invalid, keep original unchanged
-        "temp_only" - original was corrupted but temp is valid
-        "both_invalid" - both files invalid
-        False - both files were valid but empty
-    """
-    original_valid = False
-    temp_valid = False
-    original_pq = None
-    temp_pq = None
-
-    try:
-        original_pq = pq.ParquetFile(original_path)
-        original_valid = True
-    except Exception as e:
-        print(f"Warning: Original file {original_path} is corrupted or invalid: {e}", file=sys.stderr)
-
-    try:
-        if not os.path.exists(temp_path):
-            print(f"Note: Temp file {temp_path} does not exist", file=sys.stderr)
-        else:
-            temp_pq = pq.ParquetFile(temp_path)
-            temp_valid = True
-    except Exception:
-        print(f"Note: No new data in temp file {temp_path}", file=sys.stderr)
-
-    if not original_valid and not temp_valid:
-        print(f"Both original and temp files are invalid, will start fresh", file=sys.stderr)
-        return "both_invalid"
-
-    if not original_valid and temp_valid:
-        print(f"Original file corrupted but temp file is valid, recovering from temp", file=sys.stderr)
-        return "temp_only"
-
-    if original_valid and not temp_valid:
-        return "original_only"
-
-    merged_writer = None
-
-    for i in range(original_pq.num_row_groups):
-        row_group = original_pq.read_row_group(i)
-        if merged_writer is None:
-            merged_writer = pq.ParquetWriter(
-                merged_path,
-                row_group.schema,
-                flavor="spark"
-            )
-        merged_writer.write_table(row_group)
-
-    for i in range(temp_pq.num_row_groups):
-        row_group = temp_pq.read_row_group(i)
-        if merged_writer is None:
-            merged_writer = pq.ParquetWriter(
-                merged_path,
-                row_group.schema,
-                flavor="spark"
-            )
-        merged_writer.write_table(row_group)
-
-    if merged_writer is not None:
-        merged_writer.close()
-        return "merged"
-    return False
-
-
-def merge_partitioned_namespaces(partition_dir, temp_suffix, file_filter):
-    """
-    Merge partitioned namespace directories after resume.
-
-    Returns:
-        True if at least one namespace has valid data after merge
-        False if all namespaces ended up with corrupted/deleted data
-    """
-    namespace_dirs = [d for d in os.listdir(partition_dir) if d.startswith('namespace=')]
-    had_corruption = False
-    expected_temp = file_filter + temp_suffix
-
-    for ns_dir in namespace_dirs:
-        ns_path = os.path.join(partition_dir, ns_dir)
-        temp_path = os.path.join(ns_path, expected_temp)
-
-        if not os.path.exists(temp_path):
-            continue
-
-        original_file = file_filter
-        original_path = os.path.join(ns_path, original_file)
-
-        if os.path.exists(original_path):
-            merged_path = original_path + ".merged"
-            merged = merge_parquet_files(original_path, temp_path, merged_path)
-
-            if merged == "original_only":
-                if os.path.exists(temp_path):
-                    os.remove(temp_path)
-            elif merged == "temp_only":
-                os.remove(original_path)
-                os.rename(temp_path, original_path)
-            elif merged == "both_invalid":
-                if os.path.exists(original_path):
-                    os.remove(original_path)
-                if os.path.exists(temp_path):
-                    os.remove(temp_path)
-                had_corruption = True
-            elif merged == "merged":
-                os.remove(original_path)
-                os.rename(merged_path, original_path)
-                if os.path.exists(temp_path):
-                    os.remove(temp_path)
-            else:
-                if os.path.exists(original_path):
-                    os.remove(original_path)
-                if os.path.exists(temp_path):
-                    os.remove(temp_path)
-        else:
-            try:
-                pq.ParquetFile(temp_path)
-                os.rename(temp_path, original_path)
-            except Exception:
-                if os.path.exists(temp_path):
-                    os.remove(temp_path)
-                had_corruption = True
-
-    return had_corruption
-
-
-def finalize_resume_merge(
-    original_output_file,
-    temp_output_file,
-    partition_namespaces,
-    original_partition_dir
-):
-    """
-    Finalize the resume by merging temp output with original output.
-    """
-    import shutil
-
-    print("Merging resumed data with existing output...", file=sys.stderr)
-    try:
-        if partition_namespaces and original_partition_dir is not None:
-            file_filter = os.path.basename(original_output_file)
-            merge_partitioned_namespaces(original_partition_dir, ".resume_temp", file_filter)
-            if os.path.exists(temp_output_file) and os.path.isdir(temp_output_file):
-                shutil.rmtree(temp_output_file)
-        else:
-            merged_output_file = original_output_file + ".merged"
-            merged = merge_parquet_files(original_output_file, temp_output_file, merged_output_file)
-
-            if merged == "original_only":
-                if os.path.exists(temp_output_file):
-                    os.remove(temp_output_file)
-            elif merged == "temp_only":
-                os.remove(original_output_file)
-                os.rename(temp_output_file, original_output_file)
-            elif merged == "both_invalid":
-                os.remove(original_output_file)
-                if os.path.exists(temp_output_file):
-                    os.remove(temp_output_file)
-            elif merged == "merged":
-                os.remove(original_output_file)
-                os.rename(merged_output_file, original_output_file)
-                if os.path.exists(temp_output_file):
-                    os.remove(temp_output_file)
-            else:
-                os.remove(original_output_file)
-                if os.path.exists(temp_output_file):
-                    os.remove(temp_output_file)
-
-        print("Merge complete.", file=sys.stderr)
-    except Exception as e:
-        print(f"Error merging resume data for {original_output_file}: {e}", file=sys.stderr)
-        print(f"New data saved in: {temp_output_file}", file=sys.stderr)
-        raise
-
-
-def setup_resume_temp_output(output_file, partition_namespaces):
-    """
-    Set up temp output for resume mode (Parquet only).
-
-    Returns:
-        Tuple of (original_output_file, temp_output_file, original_partition_dir)
-        or (None, None, None) if no existing output to resume from.
-    """
-    import shutil
-
-    original_output_file = None
-    temp_output_file = None
-    original_partition_dir = None
-
-    if partition_namespaces:
-        partition_dir = os.path.dirname(output_file)
-        output_filename = os.path.basename(output_file)
-        output_exists = False
-        if os.path.isdir(partition_dir):
-            for d in os.listdir(partition_dir):
-                if d.startswith('namespace='):
-                    if os.path.exists(os.path.join(partition_dir, d, output_filename)):
-                        output_exists = True
-                        break
-        if output_exists:
-            original_partition_dir = partition_dir
-    else:
-        output_exists = isinstance(output_file, str) and os.path.exists(output_file)
-
-    if output_exists:
-        original_output_file = output_file
-        temp_output_file = output_file + ".resume_temp"
-
-        if os.path.exists(temp_output_file):
-            if os.path.isdir(temp_output_file):
-                shutil.rmtree(temp_output_file)
-            else:
-                os.remove(temp_output_file)
-
-        if partition_namespaces:
-            os.makedirs(temp_output_file, exist_ok=True)
-
-    return original_output_file, temp_output_file, original_partition_dir
+    result = get_jsonl_resume_point(output_file, input_file)
+    if result:
+        print(f"Resume point found from JSONL: pageid={result[0]}, revid={result[1]}", file=sys.stderr)
+    return result

src/wikiq/tables.py

@@ -1,3 +1,20 @@
+# Copyright (C) 2015-2026 Benjamin Mako Hill, Nathan TeBlunthuis, Will
+# Beason, Sohyeon Hwang, Kaylea Champion, and other contributors
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation, either version 3 of the License, or (at your
+# option) any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program. If not, see <https://www.gnu.org/licenses/>.
+
+import re
 from abc import abstractmethod, ABC
 from datetime import datetime, timezone
 from hashlib import sha1
@@ -122,6 +139,12 @@ class RevisionEditorText(RevisionField[Union[str, None]]):
 
 
 class RevisionPageTitle(RevisionField[str]):
+    """The page's title as of the time of export.
+
+    This is a page-level identity value, not a historical fact about the
+    revision: a moved page's entire history carries its final title.
+    """
+
     field = pa.field("title", pa.string())
 
     def extract(self, page: mwtypes.Page, revisions: list[mwxml.Revision]) -> str:
@@ -137,17 +160,73 @@ class RevisionDeleted(RevisionField[bool]):
 
 
 class RevisionNamespace(RevisionField[int]):
+    """The page's namespace as of the time of export.
+
+    Like the title it derives from, this is a page-level identity value
+    as of export, not a historical fact about the revision.
+    """
+
     field = pa.field("namespace", pa.int32())
 
     def extract(self, page: mwtypes.Page, revisions: list[mwxml.Revision]) -> int:
         return page.namespace
 
 
+class RedirectDetector:
+    """Detect a redirect directive at the start of revision text.
+
+    #REDIRECT is valid on every MediaWiki regardless of content language.
+    Localized aliases (e.g. OMDIRIGERING on Swedish wikis) are
+    additionally valid per wiki and can be supplied by the caller.
+    """
+
+    def __init__(self, aliases: Union[list[str], None] = None):
+        keywords = ["REDIRECT"] + list(aliases or [])
+        alternation = "|".join(re.escape(k) for k in keywords)
+        # target extraction stops at | (label) and # (fragment)
+        self.pattern = re.compile(
+            r"\A\s*#(?:" + alternation + r")\s*:?\s*\[\[([^\[\]|#]+)",
+            re.IGNORECASE,
+        )
+
+    def detect(self, text: str) -> tuple[bool, Union[str, None]]:
+        """Return (is_redirect, target); target is None when not a redirect."""
+        match = self.pattern.match(text)
+        if match is None:
+            return False, None
+        return True, match.group(1).strip()
+
+
+class RevisionIsRedirect(RevisionField[Union[bool, None]]):
+    """Whether this revision's own text begins with a redirect directive."""
+
+    field = pa.field("revision_is_redirect", pa.bool_(), nullable=True)
+
+    def __init__(self, detector: RedirectDetector):
+        super().__init__()
+        self.detector = detector
+
+    def extract(self, page: mwtypes.Page, revisions: list[mwxml.Revision]) -> Union[bool, None]:
+        revision = revisions[-1]
+        if revision.deleted.text or revision.text is None:
+            return None
+        return self.detector.detect(revision.text)[0]
+
+
 class RevisionRedirectTarget(RevisionField[Union[str, None]]):
-    field = pa.field("redirect_target", pa.string(), nullable=True)
+    """The link target of this revision's redirect directive, if any."""
+
+    field = pa.field("revision_redirect_target", pa.string(), nullable=True)
+
+    def __init__(self, detector: RedirectDetector):
+        super().__init__()
+        self.detector = detector
 
     def extract(self, page: mwtypes.Page, revisions: list[mwxml.Revision]) -> Union[str, None]:
-        return page.redirect
+        revision = revisions[-1]
+        if revision.deleted.text or revision.text is None:
+            return None
+        return self.detector.detect(revision.text)[1]
 
 
 class RevisionSha1(RevisionField[str]):
@@ -305,6 +384,50 @@ class RevisionHeadings(RevisionField[Union[list[dict], None]]):
         return self.wikitext_parser.extract_headings(revision.text)
 
 
+class RevisionContentChars(RevisionField[Union[int, None]]):
+    """Count the characters of information a revision carries.
+
+    The words, link labels and template parameter values -- what someone
+    who did not know wikitext would have typed. See
+    WikitextParser.content_sizes().
+    """
+
+    field = pa.field("content_chars", pa.int64(), nullable=True)
+
+    def __init__(self, wikitext_parser: "WikitextParser"):
+        super().__init__()
+        self.wikitext_parser = wikitext_parser
+
+    def extract(self, page: mwtypes.Page, revisions: list[mwxml.Revision]) -> Union[int, None]:
+        revision = revisions[-1]
+        if revision.deleted.text:
+            return None
+        sizes = self.wikitext_parser.content_sizes(revision.text)
+        return None if sizes is None else sizes[0]
+
+
+class RevisionMarkupChars(RevisionField[Union[int, None]]):
+    """Count the characters of markup wrapped around a revision's content.
+
+    Braces, pipes, parameter names, tag names and attributes, link targets
+    and comments. Together with content_chars this sums to the revision
+    length.
+    """
+
+    field = pa.field("markup_chars", pa.int64(), nullable=True)
+
+    def __init__(self, wikitext_parser: "WikitextParser"):
+        super().__init__()
+        self.wikitext_parser = wikitext_parser
+
+    def extract(self, page: mwtypes.Page, revisions: list[mwxml.Revision]) -> Union[int, None]:
+        revision = revisions[-1]
+        if revision.deleted.text:
+            return None
+        sizes = self.wikitext_parser.content_sizes(revision.text)
+        return None if sizes is None else sizes[1]
+
+
 class RevisionParserTimeout(RevisionField[bool]):
     """Track whether the wikitext parser timed out for this revision."""
 

src/wikiq/wiki_diff_matcher.py

@@ -1,3 +1,19 @@
+# Copyright (C) 2015-2026 Benjamin Mako Hill, Nathan TeBlunthuis, Will
+# Beason, Sohyeon Hwang, Kaylea Champion, and other contributors
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation, either version 3 of the License, or (at your
+# option) any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program. If not, see <https://www.gnu.org/licenses/>.
+
 import json
 import sys
 from collections import namedtuple
@@ -10,7 +26,6 @@ from mwpersistence import Token
 from sortedcontainers import SortedDict
 
 TOKENIZER = tokenizers.wikitext_split
-import pywikidiff2
 
 
 class DiffToOperationMap:
@@ -332,6 +347,11 @@ class WikiDiffMatcher:
 
     class Processor(DiffEngine.Processor):
         def __init__(self, tokenizer=None):
+            # imported here rather than at module scope so that importing
+            # wikiq does not require the pywikidiff2 C++ extension
+            from wikiq import require_pywikidiff2
+
+            pywikidiff2 = require_pywikidiff2("-p wikidiff2")
             self.tokenizer = tokenizer or TOKENIZER
             self.last_tokens = []
             self.previous_text = ""

src/wikiq/wikitext_parser.py

@@ -1,9 +1,36 @@
+# Copyright (C) 2015-2026 Benjamin Mako Hill, Nathan TeBlunthuis, Will
+# Beason, Sohyeon Hwang, Kaylea Champion, and other contributors
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation, either version 3 of the License, or (at your
+# option) any later version.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program. If not, see <https://www.gnu.org/licenses/>.
+
 """Shared wikitext parser with caching to avoid duplicate parsing."""
 from __future__ import annotations
 
 import signal
 
 import mwparserfromhell
+from mwparserfromhell.nodes import (
+    Argument,
+    Comment,
+    ExternalLink,
+    Heading,
+    HTMLEntity,
+    Tag,
+    Template,
+    Text,
+    Wikilink,
+)
 
 PARSER_TIMEOUT = 60  # seconds
 
@@ -22,6 +49,8 @@ class WikitextParser:
         self._cached_text: str | None = None
         self._cached_wikicode = None
         self.last_parse_timed_out: bool = False
+        self._sizes_text: str | None = None
+        self._cached_sizes: tuple[int, int] | None = None
 
     def _timeout_handler(self, signum, frame):
         raise TimeoutError("mwparserfromhell parse exceeded timeout")
@@ -130,6 +159,85 @@ class WikitextParser:
         except Exception:
             return None
 
+    def _count_content(self, wikicode, counts: list[int]) -> None:
+        """Walk a wikicode tree adding the length of rendered content.
+
+        counts is a one-element list holding the running content total.
+        Anything not counted here is markup; content_sizes() derives the
+        markup total by subtraction, so every character lands in one column
+        or the other.
+        """
+        for node in wikicode.nodes:
+            if isinstance(node, Text):
+                counts[0] += len(node.value)
+            elif isinstance(node, HTMLEntity):
+                # an entity stands in for the character it renders as, so
+                # only that character is content and the escaping is markup
+                counts[0] += len(node.normalize())
+            elif isinstance(node, Comment):
+                # never reaches a reader, so it is not information
+                continue
+            elif isinstance(node, Template):
+                # the parameter values are the information; the template
+                # name, parameter names and delimiters are the polish that
+                # someone later wrapped around it
+                for param in node.params:
+                    self._count_content(param.value, counts)
+            elif isinstance(node, Tag):
+                # tag contents are information; the tag name and attributes
+                # are markup. This covers ''italic'' and '''bold''', which
+                # mwparserfromhell represents as i and b tags
+                if node.contents is not None:
+                    self._count_content(node.contents, counts)
+            elif isinstance(node, Wikilink):
+                # the label if present, otherwise the target, is what a
+                # reader sees; the brackets and the link target are markup
+                rendered = node.text if node.text is not None else node.title
+                self._count_content(rendered, counts)
+            elif isinstance(node, ExternalLink):
+                # both halves are information someone typed: a contributor
+                # who did not know wikitext would still have written the URL
+                counts[0] += len(str(node.url))
+                if node.title is not None:
+                    self._count_content(node.title, counts)
+            elif isinstance(node, Heading):
+                self._count_content(node.title, counts)
+            elif isinstance(node, Argument):
+                if node.default is not None:
+                    self._count_content(node.default, counts)
+
+    def content_sizes(self, text: str | None) -> tuple[int, int] | None:
+        """Split a revision into the information it carries and the markup
+        around it.
+
+        Returns (content_chars, markup_chars), which sum to len(text).
+
+        Content is what a contributor who did not know wikitext would have
+        typed: the words, the link labels, the values inside template
+        parameters. Markup is everything a later editor added to format it —
+        braces, pipes, parameter names, tag names and attributes, link
+        targets, quote marks, and comments.
+
+        So "Wonderful Hotel, +15555550123" is almost all content, while
+        "{{listing|name=Wonderful Hotel|phone=+15555550123}}" carries the
+        same information with two dozen characters of markup around it.
+        """
+        if text is None:
+            return None
+        if text == self._sizes_text:
+            return self._cached_sizes
+        try:
+            wikicode = self._get_wikicode(text)
+            counts = [0]
+            self._count_content(wikicode, counts)
+            content = min(counts[0], len(text))
+            result = (content, len(text) - content)
+        except Exception:
+            result = None
+        self._sizes_text = text
+        self._cached_sizes = result
+        return result
+
     def extract_headings(self, text: str | None) -> list[dict] | None:
         """Extract all section headings with their levels."""
         if text is None:

test/baseline_output/basic_regextest_0.tsv

@@ -1,27 +1,27 @@
-"revid"	"date_time"	"articleid"	"title"	"namespace"	"deleted"	"editorid"	"edit_summary"	"text_chars"	"reverteds"	"sha1"	"minor"	"editor"	"anon"	"revert"	"threedigits"
-819091731	2018-01-07 10:40:58	56237363	"User talk:86.139.142.254"	3	false	3742946	"Your IP address has been blocked from editing because it has been used to [[WP:EVADE|evade a previous block]]. ([[WP:TW|TW]])"	1141		"135nz8q6lfam6cojla7azb7k5alx3t3"	false	"NinjaRobotPirate"	false	false	"126, 126, 126, 126"
-819091755	2018-01-07 10:41:10	56237364	"User talk:Kavin kavitha"	3	false	32792125	"[[WP:AES|←]]Created page with ''''''Kavin (Tamil. கவின்) is a masculine given name, which is Tamil for ""beauty"", ""grace"", ""fairness"" or ""comeliness""Kavin is born on 01 /12/2001 at Sa...'"	663		"0pwezjc6yopz0smc8al6ogc4fax5bwo"	false	"Kavin kavitha"	false	false	
-819091788	2018-01-07 10:41:26	56237365	"User talk:Dr.vivek163"	3	false	32621254	"/* Regarding Merger discussion */ new section"	399		"sz3t2ap7z8bpkdvdvi195f3i35949bv"	false	"Amicable always"	false	false	
-819091796	2018-01-07 10:41:31	56237366	"User talk:Twistorl"	3	false	13286072	"Warning [[Special:Contributions/Twistorl|Twistorl]] - #1"	1260		"r6s5j8j3iykenrhuhpnkpsmmd71vubf"	false	"ClueBot NG"	false	false	
-819091825	2018-01-07 10:41:51	56237368	"Kom Firin"	0	false	8409334	"[[WP:AES|←]]Created page with '[[File:Stele 67.119 Brooklyn.jpg|thumb|Stele of the [[Libu#Great Chiefs of the Libu|Chief of the Libu]] Titaru, a contemporary of pharaoh [[Shoshenq V]] of the [...'TestCaseB and you're a Tor node "	2249		"tf5qz2yaswx61zrlm9ovxzuhl7r2dc4"	false	"Khruner"	false	false	"119, 978, 500, 292, 225, 199, 292"
-822610647	2018-01-27 12:16:02	56237368	"Kom Firin"	0	false	8409334	"/* History */ typo"	2230		"e6oa4g0qv64icdaq26uu1zzbyr5hcbh"	true	"Khruner"	false	false	"119, 978, 500, 292, 225, 199, 292"
-819091844	2018-01-07 10:42:05	56237369	"User:Editingaccount1994/sandbox"	2	false	32794215	"[[WP:AES|←]]Created page with '{{User sandbox}} <!-- EDIT BELOW THIS LINE --> {{voir homonymes|Chevalier}} {{Infobox Artiste  | nom                = Li Chevalier  | autres noms        =   | im...'"	27840		"0fyvyh2a8xu41gt8obr34oba0bfixj6"	false	"Editingaccount1994"	false	false	"798, 150, 150, 150, 621, 137, 137, 150, 150, 350, 195, 350, 195, 180, 180, 350, 195, 300, 150, 150, 150, 180, 180, 621"
-819093984	2018-01-07 11:09:52	56237369	"User:Editingaccount1994/sandbox"	2	false	7611264	"[[User:AnomieBOT/docs/TemplateSubster|Substing templates]]: {{Lien web}}. See [[User:AnomieBOT/docs/TemplateSubster]] for info."	27787		"8gy52aolt5rg3eaketwj5v7eiw0apv2"	true	"AnomieBOT"	false	false	"798, 150, 150, 150, 621, 137, 137, 150, 150, 350, 195, 350, 195, 180, 180, 350, 195, 300, 150, 150, 150, 180, 180, 621"
-820064189	2018-01-12 21:45:50	56237369	"User:Editingaccount1994/sandbox"	2	false	12406635	"Orphan per [[WP:TFD|TFD outcome]]"	27784		"he8ydemaanxlrpftqxkez8jfpge1fsj"	true	"SporkBot"	false	false	"798, 150, 150, 150, 621, 137, 137, 150, 150, 350, 195, 350, 195, 180, 180, 350, 195, 300, 150, 150, 150, 180, 180, 621"
-820078679	2018-01-12 23:28:11	56237369	"User:Editingaccount1994/sandbox"	2	false	12406635	"Replace template per [[Wikipedia:Templates for discussion/Log/2010 June 13|TFD outcome]]; no change in content"	27783		"0to17w9rth3url8n7gvucdtobybdq5h"	true	"SporkBot"	false	false	"798, 150, 150, 150, 621, 137, 137, 150, 150, 350, 195, 350, 195, 180, 180, 350, 195, 300, 150, 150, 150, 180, 180, 621"
-820078733	2018-01-12 23:28:39	56237369	"User:Editingaccount1994/sandbox"	2	false	12406635	"Replace template per [[Wikipedia:Templates for discussion/Log/2011 February 17|TFD outcome]]; no change in content"	27782		"531dizmmloyxffbkdr5vph7owh921eg"	true	"SporkBot"	false	false	"798, 150, 150, 150, 621, 137, 137, 150, 150, 350, 195, 350, 195, 180, 180, 350, 195, 300, 150, 150, 150, 180, 180, 621"
-820177382	2018-01-13 13:45:33	56237369	"User:Editingaccount1994/sandbox"	2	false	13791031	"translate TestCaseD if you are from tor you need neutral point of view "	27757		"nik9p2u2fuk4yazjxt8ymbicxv5qid9"	false	"Frietjes"	false	false	"798, 150, 150, 150, 621, 100, 621"
-822038928	2018-01-24 01:35:22	56237369	"User:Editingaccount1994/sandbox"	2	false	2304267	"Removing [[:c:File:Li_Chevalier_Art_Studio.jpg|Li_Chevalier_Art_Studio.jpg]], it has been deleted from Commons by [[:c:User:JuTa|JuTa]] because: [[:c:COM:OTRS|No permission]] since 16 January 2018."	27667		"gwk6pampl8si1v5pv3kwgteg710sfw3"	false	"CommonsDelinker"	false	false	"798, 150, 150, 150, 621, 100, 621"
-819091874	2018-01-07 10:42:20	56237370	"Anita del Rey"	0	false	1368779	"r from alt name"	25		"n4ozbsgle13p9yywtfrz982ccj8woc9"	false	"PamD"	false	false	
-819091883	2018-01-07 10:42:27	56237371	"User talk:119.94.96.157"	3	false	13286072	"Warning [[Special:Contributions/119.94.96.157|119.94.96.157]] - #1"	1274		"ksohnvsbeuzwpl5vb8a3v8m18hva0a7"	false	"ClueBot NG"	false	false	"119, 157, 119, 157, 119, 157, 119, 157"
-819091914	2018-01-07 10:42:50	56237372	"Category:Ohmi Railway"	14	false	677153	"[[WP:AES|←]]Created page with ' [[Category:Railway companies of Japan]] [[Category:Rail transport in Shiga Prefecture]] [[Category:Seibu Group]]'"	113		"je7aw21fedbwyqsyofpisdrynsu7olr"	false	"Underbar dk"	false	false	
-819091968	2018-01-07 10:43:32	56237375	"User talk:92.226.219.222"	3	false	882433	"[[WP:AES|←]]Created page with '{{3rr}}~~~~'"	199		"cpm4tkzcx4hc6irr9ukbi06ogud8dtq"	false	"TastyPoutine"	false	false	
-819094036	2018-01-07 11:10:24	56237375	"User talk:92.226.219.222"	3	false	7611264	"[[User:AnomieBOT/docs/TemplateSubster|Substing templates]]: {{3rr}}. See [[User:AnomieBOT/docs/TemplateSubster]] for info."	1840		"artmfz8b2gxhb3pp8a5p4ksplxqfkpg"	true	"AnomieBOT"	false	false	
-819112363	2018-01-07 14:33:36	56237375	"User talk:92.226.219.222"	3	false	702940	"Your IP address has been blocked from editing because it has been used to [[WP:EVADE|evade a previous block]]. ([[WP:TW|TW]])"	2949		"dn9wj0n8d8pdd5lqe56uw5xamupowr1"	false	"Only"	false	false	"126, 126, 126, 126"
-819092004	2018-01-07 10:44:01	56237376	"User:Dipayanacharya"	2	false	32794237	"Education"	28		"ofueugwatmmn7u73isw732neuza57gk"	false	"Dipayanacharya"	false	false	
-819092390	2018-01-07 10:49:08	56237376	"User:Dipayanacharya"	2	false	32794237	"School"	38		"dsz55xv96ec2uv6w9c1z7c52ipfovbw"	false	"Dipayanacharya"	false	false	
-819092066	2018-01-07 10:44:56	56237378	"BSCIC"	0	false	21516552	"[[WP:AES|←]]Redirected page to [[Bangladesh Small and Cottage Industries Corporation]]"	65		"9ma38hak0ef1ew4fpiutxpnzd8oz1wd"	false	"Vinegarymass911"	false	false	
-819092102	2018-01-07 10:45:21	56237379	"Category:Women government ministers of Yemen"	14	false	754619	"[[WP:AES|←]]Created page with '{{portal|Yemen|Politics}} {{Non-diffusing subcategory|Government ministers of Yemen}}  {{Underpopulated category}}  Category:Women government ministers by nati...'"	285		"4dvakoat58bzyf5hmtthxukt29hip6n"	false	"BrownHairedGirl"	false	false	
-819092135	2018-01-07 10:45:54	56237381	"Talk:List of Morning Glories Characters"	1	false	410898	"[[WP:AES|←]]Created page with '{{WikiProject Fictional characters|class=List|importance=low}} {{Comicsproj|class=List|importance=low}}'"	103		"2sjrxsc7os9k9pg4su2t4rk2j8nn0h7"	false	"PRehse"	false	false	
-819092138	2018-01-07 10:45:56	56237382	"User talk:106.207.126.114"	3	false	13286072	"Warning [[Special:Contributions/106.207.126.114|106.207.126.114]] - #1"	1330		"3y9t5wpk6ur5jhone75rhm4wjf01fgi"	false	"ClueBot NG"	false	false	"106, 207, 126, 114, 106, 207, 126, 114, 106, 207, 126, 114, 106, 207, 126, 114"
-819092495	2018-01-07 10:50:22	56237382	"User talk:106.207.126.114"	3	false	31190506	"Caution: Unconstructive editing on [[List of Baahubali characters]]. ([[WP:TW|TW]])"	2355		"8wvn6vh3isyt0dorpe89lztrburgupe"	false	"HindWIKI"	false	false	"106, 207, 126, 114, 106, 207, 126, 114, 106, 207, 126, 114, 106, 207, 126, 114"
+"revid"	"date_time"	"articleid"	"title"	"namespace"	"revision_is_redirect"	"revision_redirect_target"	"deleted"	"editorid"	"edit_summary"	"text_chars"	"reverteds"	"sha1"	"minor"	"editor"	"anon"	"revert"	"threedigits"
+819091731	2018-01-07 10:40:58	56237363	"User talk:86.139.142.254"	3	false		false	3742946	"Your IP address has been blocked from editing because it has been used to [[WP:EVADE|evade a previous block]]. ([[WP:TW|TW]])"	1141		"135nz8q6lfam6cojla7azb7k5alx3t3"	false	"NinjaRobotPirate"	false	false	"126, 126, 126, 126"
+819091755	2018-01-07 10:41:10	56237364	"User talk:Kavin kavitha"	3	false		false	32792125	"[[WP:AES|←]]Created page with ''''''Kavin (Tamil. கவின்) is a masculine given name, which is Tamil for ""beauty"", ""grace"", ""fairness"" or ""comeliness""Kavin is born on 01 /12/2001 at Sa...'"	663		"0pwezjc6yopz0smc8al6ogc4fax5bwo"	false	"Kavin kavitha"	false	false	
+819091788	2018-01-07 10:41:26	56237365	"User talk:Dr.vivek163"	3	false		false	32621254	"/* Regarding Merger discussion */ new section"	399		"sz3t2ap7z8bpkdvdvi195f3i35949bv"	false	"Amicable always"	false	false	
+819091796	2018-01-07 10:41:31	56237366	"User talk:Twistorl"	3	false		false	13286072	"Warning [[Special:Contributions/Twistorl|Twistorl]] - #1"	1260		"r6s5j8j3iykenrhuhpnkpsmmd71vubf"	false	"ClueBot NG"	false	false	
+819091825	2018-01-07 10:41:51	56237368	"Kom Firin"	0	false		false	8409334	"[[WP:AES|←]]Created page with '[[File:Stele 67.119 Brooklyn.jpg|thumb|Stele of the [[Libu#Great Chiefs of the Libu|Chief of the Libu]] Titaru, a contemporary of pharaoh [[Shoshenq V]] of the [...'TestCaseB and you're a Tor node "	2249		"tf5qz2yaswx61zrlm9ovxzuhl7r2dc4"	false	"Khruner"	false	false	"119, 978, 500, 292, 225, 199, 292"
+822610647	2018-01-27 12:16:02	56237368	"Kom Firin"	0	false		false	8409334	"/* History */ typo"	2230		"e6oa4g0qv64icdaq26uu1zzbyr5hcbh"	true	"Khruner"	false	false	"119, 978, 500, 292, 225, 199, 292"
+819091844	2018-01-07 10:42:05	56237369	"User:Editingaccount1994/sandbox"	2	false		false	32794215	"[[WP:AES|←]]Created page with '{{User sandbox}} <!-- EDIT BELOW THIS LINE --> {{voir homonymes|Chevalier}} {{Infobox Artiste  | nom                = Li Chevalier  | autres noms        =   | im...'"	27840		"0fyvyh2a8xu41gt8obr34oba0bfixj6"	false	"Editingaccount1994"	false	false	"798, 150, 150, 150, 621, 137, 137, 150, 150, 350, 195, 350, 195, 180, 180, 350, 195, 300, 150, 150, 150, 180, 180, 621"
+819093984	2018-01-07 11:09:52	56237369	"User:Editingaccount1994/sandbox"	2	false		false	7611264	"[[User:AnomieBOT/docs/TemplateSubster|Substing templates]]: {{Lien web}}. See [[User:AnomieBOT/docs/TemplateSubster]] for info."	27787		"8gy52aolt5rg3eaketwj5v7eiw0apv2"	true	"AnomieBOT"	false	false	"798, 150, 150, 150, 621, 137, 137, 150, 150, 350, 195, 350, 195, 180, 180, 350, 195, 300, 150, 150, 150, 180, 180, 621"
+820064189	2018-01-12 21:45:50	56237369	"User:Editingaccount1994/sandbox"	2	false		false	12406635	"Orphan per [[WP:TFD|TFD outcome]]"	27784		"he8ydemaanxlrpftqxkez8jfpge1fsj"	true	"SporkBot"	false	false	"798, 150, 150, 150, 621, 137, 137, 150, 150, 350, 195, 350, 195, 180, 180, 350, 195, 300, 150, 150, 150, 180, 180, 621"
+820078679	2018-01-12 23:28:11	56237369	"User:Editingaccount1994/sandbox"	2	false		false	12406635	"Replace template per [[Wikipedia:Templates for discussion/Log/2010 June 13|TFD outcome]]; no change in content"	27783		"0to17w9rth3url8n7gvucdtobybdq5h"	true	"SporkBot"	false	false	"798, 150, 150, 150, 621, 137, 137, 150, 150, 350, 195, 350, 195, 180, 180, 350, 195, 300, 150, 150, 150, 180, 180, 621"
+820078733	2018-01-12 23:28:39	56237369	"User:Editingaccount1994/sandbox"	2	false		false	12406635	"Replace template per [[Wikipedia:Templates for discussion/Log/2011 February 17|TFD outcome]]; no change in content"	27782		"531dizmmloyxffbkdr5vph7owh921eg"	true	"SporkBot"	false	false	"798, 150, 150, 150, 621, 137, 137, 150, 150, 350, 195, 350, 195, 180, 180, 350, 195, 300, 150, 150, 150, 180, 180, 621"
+820177382	2018-01-13 13:45:33	56237369	"User:Editingaccount1994/sandbox"	2	false		false	13791031	"translate TestCaseD if you are from tor you need neutral point of view "	27757		"nik9p2u2fuk4yazjxt8ymbicxv5qid9"	false	"Frietjes"	false	false	"798, 150, 150, 150, 621, 100, 621"
+822038928	2018-01-24 01:35:22	56237369	"User:Editingaccount1994/sandbox"	2	false		false	2304267	"Removing [[:c:File:Li_Chevalier_Art_Studio.jpg|Li_Chevalier_Art_Studio.jpg]], it has been deleted from Commons by [[:c:User:JuTa|JuTa]] because: [[:c:COM:OTRS|No permission]] since 16 January 2018."	27667		"gwk6pampl8si1v5pv3kwgteg710sfw3"	false	"CommonsDelinker"	false	false	"798, 150, 150, 150, 621, 100, 621"
+819091874	2018-01-07 10:42:20	56237370	"Anita del Rey"	0	true	"Ana del Rey"	false	1368779	"r from alt name"	25		"n4ozbsgle13p9yywtfrz982ccj8woc9"	false	"PamD"	false	false	
+819091883	2018-01-07 10:42:27	56237371	"User talk:119.94.96.157"	3	false		false	13286072	"Warning [[Special:Contributions/119.94.96.157|119.94.96.157]] - #1"	1274		"ksohnvsbeuzwpl5vb8a3v8m18hva0a7"	false	"ClueBot NG"	false	false	"119, 157, 119, 157, 119, 157, 119, 157"
+819091914	2018-01-07 10:42:50	56237372	"Category:Ohmi Railway"	14	false		false	677153	"[[WP:AES|←]]Created page with ' [[Category:Railway companies of Japan]] [[Category:Rail transport in Shiga Prefecture]] [[Category:Seibu Group]]'"	113		"je7aw21fedbwyqsyofpisdrynsu7olr"	false	"Underbar dk"	false	false	
+819091968	2018-01-07 10:43:32	56237375	"User talk:92.226.219.222"	3	false		false	882433	"[[WP:AES|←]]Created page with '{{3rr}}~~~~'"	199		"cpm4tkzcx4hc6irr9ukbi06ogud8dtq"	false	"TastyPoutine"	false	false	
+819094036	2018-01-07 11:10:24	56237375	"User talk:92.226.219.222"	3	false		false	7611264	"[[User:AnomieBOT/docs/TemplateSubster|Substing templates]]: {{3rr}}. See [[User:AnomieBOT/docs/TemplateSubster]] for info."	1840		"artmfz8b2gxhb3pp8a5p4ksplxqfkpg"	true	"AnomieBOT"	false	false	
+819112363	2018-01-07 14:33:36	56237375	"User talk:92.226.219.222"	3	false		false	702940	"Your IP address has been blocked from editing because it has been used to [[WP:EVADE|evade a previous block]]. ([[WP:TW|TW]])"	2949		"dn9wj0n8d8pdd5lqe56uw5xamupowr1"	false	"Only"	false	false	"126, 126, 126, 126"
+819092004	2018-01-07 10:44:01	56237376	"User:Dipayanacharya"	2	false		false	32794237	"Education"	28		"ofueugwatmmn7u73isw732neuza57gk"	false	"Dipayanacharya"	false	false	
+819092390	2018-01-07 10:49:08	56237376	"User:Dipayanacharya"	2	false		false	32794237	"School"	38		"dsz55xv96ec2uv6w9c1z7c52ipfovbw"	false	"Dipayanacharya"	false	false	
+819092066	2018-01-07 10:44:56	56237378	"BSCIC"	0	true	"Bangladesh Small and Cottage Industries Corporation"	false	21516552	"[[WP:AES|←]]Redirected page to [[Bangladesh Small and Cottage Industries Corporation]]"	65		"9ma38hak0ef1ew4fpiutxpnzd8oz1wd"	false	"Vinegarymass911"	false	false	
+819092102	2018-01-07 10:45:21	56237379	"Category:Women government ministers of Yemen"	14	false		false	754619	"[[WP:AES|←]]Created page with '{{portal|Yemen|Politics}} {{Non-diffusing subcategory|Government ministers of Yemen}}  {{Underpopulated category}}  Category:Women government ministers by nati...'"	285		"4dvakoat58bzyf5hmtthxukt29hip6n"	false	"BrownHairedGirl"	false	false	
+819092135	2018-01-07 10:45:54	56237381	"Talk:List of Morning Glories Characters"	1	false		false	410898	"[[WP:AES|←]]Created page with '{{WikiProject Fictional characters|class=List|importance=low}} {{Comicsproj|class=List|importance=low}}'"	103		"2sjrxsc7os9k9pg4su2t4rk2j8nn0h7"	false	"PRehse"	false	false	
+819092138	2018-01-07 10:45:56	56237382	"User talk:106.207.126.114"	3	false		false	13286072	"Warning [[Special:Contributions/106.207.126.114|106.207.126.114]] - #1"	1330		"3y9t5wpk6ur5jhone75rhm4wjf01fgi"	false	"ClueBot NG"	false	false	"106, 207, 126, 114, 106, 207, 126, 114, 106, 207, 126, 114, 106, 207, 126, 114"
+819092495	2018-01-07 10:50:22	56237382	"User talk:106.207.126.114"	3	false		false	31190506	"Caution: Unconstructive editing on [[List of Baahubali characters]]. ([[WP:TW|TW]])"	2355		"8wvn6vh3isyt0dorpe89lztrburgupe"	false	"HindWIKI"	false	false	"106, 207, 126, 114, 106, 207, 126, 114, 106, 207, 126, 114, 106, 207, 126, 114"

test/baseline_output/basic_regextest_1.tsv

@@ -1,27 +1,27 @@
-"revid"	"date_time"	"articleid"	"title"	"namespace"	"deleted"	"editorid"	"edit_summary"	"text_chars"	"reverteds"	"sha1"	"minor"	"editor"	"anon"	"revert"	"testcases"	"page_word"
-819091731	2018-01-07 10:40:58	56237363	"User talk:86.139.142.254"	3	false	3742946	"Your IP address has been blocked from editing because it has been used to [[WP:EVADE|evade a previous block]]. ([[WP:TW|TW]])"	1141		"135nz8q6lfam6cojla7azb7k5alx3t3"	false	"NinjaRobotPirate"	false	false		"page, page"
-819091755	2018-01-07 10:41:10	56237364	"User talk:Kavin kavitha"	3	false	32792125	"[[WP:AES|←]]Created page with ''''''Kavin (Tamil. கவின்) is a masculine given name, which is Tamil for ""beauty"", ""grace"", ""fairness"" or ""comeliness""Kavin is born on 01 /12/2001 at Sa...'"	663		"0pwezjc6yopz0smc8al6ogc4fax5bwo"	false	"Kavin kavitha"	false	false		
-819091788	2018-01-07 10:41:26	56237365	"User talk:Dr.vivek163"	3	false	32621254	"/* Regarding Merger discussion */ new section"	399		"sz3t2ap7z8bpkdvdvi195f3i35949bv"	false	"Amicable always"	false	false	"TestCase, TestCase"	
-819091796	2018-01-07 10:41:31	56237366	"User talk:Twistorl"	3	false	13286072	"Warning [[Special:Contributions/Twistorl|Twistorl]] - #1"	1260		"r6s5j8j3iykenrhuhpnkpsmmd71vubf"	false	"ClueBot NG"	false	false		"page"
-819091825	2018-01-07 10:41:51	56237368	"Kom Firin"	0	false	8409334	"[[WP:AES|←]]Created page with '[[File:Stele 67.119 Brooklyn.jpg|thumb|Stele of the [[Libu#Great Chiefs of the Libu|Chief of the Libu]] Titaru, a contemporary of pharaoh [[Shoshenq V]] of the [...'TestCaseB and you're a Tor node "	2249		"tf5qz2yaswx61zrlm9ovxzuhl7r2dc4"	false	"Khruner"	false	false	"TestCase"	"page"
-822610647	2018-01-27 12:16:02	56237368	"Kom Firin"	0	false	8409334	"/* History */ typo"	2230		"e6oa4g0qv64icdaq26uu1zzbyr5hcbh"	true	"Khruner"	false	false		"page"
-819091844	2018-01-07 10:42:05	56237369	"User:Editingaccount1994/sandbox"	2	false	32794215	"[[WP:AES|←]]Created page with '{{User sandbox}} <!-- EDIT BELOW THIS LINE --> {{voir homonymes|Chevalier}} {{Infobox Artiste  | nom                = Li Chevalier  | autres noms        =   | im...'"	27840		"0fyvyh2a8xu41gt8obr34oba0bfixj6"	false	"Editingaccount1994"	false	false		"page, page"
-819093984	2018-01-07 11:09:52	56237369	"User:Editingaccount1994/sandbox"	2	false	7611264	"[[User:AnomieBOT/docs/TemplateSubster|Substing templates]]: {{Lien web}}. See [[User:AnomieBOT/docs/TemplateSubster]] for info."	27787		"8gy52aolt5rg3eaketwj5v7eiw0apv2"	true	"AnomieBOT"	false	false		"page, page"
-820064189	2018-01-12 21:45:50	56237369	"User:Editingaccount1994/sandbox"	2	false	12406635	"Orphan per [[WP:TFD|TFD outcome]]"	27784		"he8ydemaanxlrpftqxkez8jfpge1fsj"	true	"SporkBot"	false	false		"page, page"
-820078679	2018-01-12 23:28:11	56237369	"User:Editingaccount1994/sandbox"	2	false	12406635	"Replace template per [[Wikipedia:Templates for discussion/Log/2010 June 13|TFD outcome]]; no change in content"	27783		"0to17w9rth3url8n7gvucdtobybdq5h"	true	"SporkBot"	false	false		"page, page"
-820078733	2018-01-12 23:28:39	56237369	"User:Editingaccount1994/sandbox"	2	false	12406635	"Replace template per [[Wikipedia:Templates for discussion/Log/2011 February 17|TFD outcome]]; no change in content"	27782		"531dizmmloyxffbkdr5vph7owh921eg"	true	"SporkBot"	false	false		"page, page"
-820177382	2018-01-13 13:45:33	56237369	"User:Editingaccount1994/sandbox"	2	false	13791031	"translate TestCaseD if you are from tor you need neutral point of view "	27757		"nik9p2u2fuk4yazjxt8ymbicxv5qid9"	false	"Frietjes"	false	false		"page, page"
-822038928	2018-01-24 01:35:22	56237369	"User:Editingaccount1994/sandbox"	2	false	2304267	"Removing [[:c:File:Li_Chevalier_Art_Studio.jpg|Li_Chevalier_Art_Studio.jpg]], it has been deleted from Commons by [[:c:User:JuTa|JuTa]] because: [[:c:COM:OTRS|No permission]] since 16 January 2018."	27667		"gwk6pampl8si1v5pv3kwgteg710sfw3"	false	"CommonsDelinker"	false	false		"page, page"
-819091874	2018-01-07 10:42:20	56237370	"Anita del Rey"	0	false	1368779	"r from alt name"	25		"n4ozbsgle13p9yywtfrz982ccj8woc9"	false	"PamD"	false	false		
-819091883	2018-01-07 10:42:27	56237371	"User talk:119.94.96.157"	3	false	13286072	"Warning [[Special:Contributions/119.94.96.157|119.94.96.157]] - #1"	1274		"ksohnvsbeuzwpl5vb8a3v8m18hva0a7"	false	"ClueBot NG"	false	false		"page"
-819091914	2018-01-07 10:42:50	56237372	"Category:Ohmi Railway"	14	false	677153	"[[WP:AES|←]]Created page with ' [[Category:Railway companies of Japan]] [[Category:Rail transport in Shiga Prefecture]] [[Category:Seibu Group]]'"	113		"je7aw21fedbwyqsyofpisdrynsu7olr"	false	"Underbar dk"	false	false		
-819091968	2018-01-07 10:43:32	56237375	"User talk:92.226.219.222"	3	false	882433	"[[WP:AES|←]]Created page with '{{3rr}}~~~~'"	199		"cpm4tkzcx4hc6irr9ukbi06ogud8dtq"	false	"TastyPoutine"	false	false		
-819094036	2018-01-07 11:10:24	56237375	"User talk:92.226.219.222"	3	false	7611264	"[[User:AnomieBOT/docs/TemplateSubster|Substing templates]]: {{3rr}}. See [[User:AnomieBOT/docs/TemplateSubster]] for info."	1840		"artmfz8b2gxhb3pp8a5p4ksplxqfkpg"	true	"AnomieBOT"	false	false		"page, page, page, page"
-819112363	2018-01-07 14:33:36	56237375	"User talk:92.226.219.222"	3	false	702940	"Your IP address has been blocked from editing because it has been used to [[WP:EVADE|evade a previous block]]. ([[WP:TW|TW]])"	2949		"dn9wj0n8d8pdd5lqe56uw5xamupowr1"	false	"Only"	false	false		"page, page, page, page, page, page"
-819092004	2018-01-07 10:44:01	56237376	"User:Dipayanacharya"	2	false	32794237	"Education"	28		"ofueugwatmmn7u73isw732neuza57gk"	false	"Dipayanacharya"	false	false		
-819092390	2018-01-07 10:49:08	56237376	"User:Dipayanacharya"	2	false	32794237	"School"	38		"dsz55xv96ec2uv6w9c1z7c52ipfovbw"	false	"Dipayanacharya"	false	false		
-819092066	2018-01-07 10:44:56	56237378	"BSCIC"	0	false	21516552	"[[WP:AES|←]]Redirected page to [[Bangladesh Small and Cottage Industries Corporation]]"	65		"9ma38hak0ef1ew4fpiutxpnzd8oz1wd"	false	"Vinegarymass911"	false	false		
-819092102	2018-01-07 10:45:21	56237379	"Category:Women government ministers of Yemen"	14	false	754619	"[[WP:AES|←]]Created page with '{{portal|Yemen|Politics}} {{Non-diffusing subcategory|Government ministers of Yemen}}  {{Underpopulated category}}  Category:Women government ministers by nati...'"	285		"4dvakoat58bzyf5hmtthxukt29hip6n"	false	"BrownHairedGirl"	false	false		
-819092135	2018-01-07 10:45:54	56237381	"Talk:List of Morning Glories Characters"	1	false	410898	"[[WP:AES|←]]Created page with '{{WikiProject Fictional characters|class=List|importance=low}} {{Comicsproj|class=List|importance=low}}'"	103		"2sjrxsc7os9k9pg4su2t4rk2j8nn0h7"	false	"PRehse"	false	false		
-819092138	2018-01-07 10:45:56	56237382	"User talk:106.207.126.114"	3	false	13286072	"Warning [[Special:Contributions/106.207.126.114|106.207.126.114]] - #1"	1330		"3y9t5wpk6ur5jhone75rhm4wjf01fgi"	false	"ClueBot NG"	false	false		"page"
-819092495	2018-01-07 10:50:22	56237382	"User talk:106.207.126.114"	3	false	31190506	"Caution: Unconstructive editing on [[List of Baahubali characters]]. ([[WP:TW|TW]])"	2355		"8wvn6vh3isyt0dorpe89lztrburgupe"	false	"HindWIKI"	false	false		"page"
+"revid"	"date_time"	"articleid"	"title"	"namespace"	"revision_is_redirect"	"revision_redirect_target"	"deleted"	"editorid"	"edit_summary"	"text_chars"	"reverteds"	"sha1"	"minor"	"editor"	"anon"	"revert"	"testcases"	"page_word"
+819091731	2018-01-07 10:40:58	56237363	"User talk:86.139.142.254"	3	false		false	3742946	"Your IP address has been blocked from editing because it has been used to [[WP:EVADE|evade a previous block]]. ([[WP:TW|TW]])"	1141		"135nz8q6lfam6cojla7azb7k5alx3t3"	false	"NinjaRobotPirate"	false	false		"page, page"
+819091755	2018-01-07 10:41:10	56237364	"User talk:Kavin kavitha"	3	false		false	32792125	"[[WP:AES|←]]Created page with ''''''Kavin (Tamil. கவின்) is a masculine given name, which is Tamil for ""beauty"", ""grace"", ""fairness"" or ""comeliness""Kavin is born on 01 /12/2001 at Sa...'"	663		"0pwezjc6yopz0smc8al6ogc4fax5bwo"	false	"Kavin kavitha"	false	false		
+819091788	2018-01-07 10:41:26	56237365	"User talk:Dr.vivek163"	3	false		false	32621254	"/* Regarding Merger discussion */ new section"	399		"sz3t2ap7z8bpkdvdvi195f3i35949bv"	false	"Amicable always"	false	false	"TestCase, TestCase"	
+819091796	2018-01-07 10:41:31	56237366	"User talk:Twistorl"	3	false		false	13286072	"Warning [[Special:Contributions/Twistorl|Twistorl]] - #1"	1260		"r6s5j8j3iykenrhuhpnkpsmmd71vubf"	false	"ClueBot NG"	false	false		"page"
+819091825	2018-01-07 10:41:51	56237368	"Kom Firin"	0	false		false	8409334	"[[WP:AES|←]]Created page with '[[File:Stele 67.119 Brooklyn.jpg|thumb|Stele of the [[Libu#Great Chiefs of the Libu|Chief of the Libu]] Titaru, a contemporary of pharaoh [[Shoshenq V]] of the [...'TestCaseB and you're a Tor node "	2249		"tf5qz2yaswx61zrlm9ovxzuhl7r2dc4"	false	"Khruner"	false	false	"TestCase"	"page"
+822610647	2018-01-27 12:16:02	56237368	"Kom Firin"	0	false		false	8409334	"/* History */ typo"	2230		"e6oa4g0qv64icdaq26uu1zzbyr5hcbh"	true	"Khruner"	false	false		"page"
+819091844	2018-01-07 10:42:05	56237369	"User:Editingaccount1994/sandbox"	2	false		false	32794215	"[[WP:AES|←]]Created page with '{{User sandbox}} <!-- EDIT BELOW THIS LINE --> {{voir homonymes|Chevalier}} {{Infobox Artiste  | nom                = Li Chevalier  | autres noms        =   | im...'"	27840		"0fyvyh2a8xu41gt8obr34oba0bfixj6"	false	"Editingaccount1994"	false	false		"page, page"
+819093984	2018-01-07 11:09:52	56237369	"User:Editingaccount1994/sandbox"	2	false		false	7611264	"[[User:AnomieBOT/docs/TemplateSubster|Substing templates]]: {{Lien web}}. See [[User:AnomieBOT/docs/TemplateSubster]] for info."	27787		"8gy52aolt5rg3eaketwj5v7eiw0apv2"	true	"AnomieBOT"	false	false		"page, page"
+820064189	2018-01-12 21:45:50	56237369	"User:Editingaccount1994/sandbox"	2	false		false	12406635	"Orphan per [[WP:TFD|TFD outcome]]"	27784		"he8ydemaanxlrpftqxkez8jfpge1fsj"	true	"SporkBot"	false	false		"page, page"
+820078679	2018-01-12 23:28:11	56237369	"User:Editingaccount1994/sandbox"	2	false		false	12406635	"Replace template per [[Wikipedia:Templates for discussion/Log/2010 June 13|TFD outcome]]; no change in content"	27783		"0to17w9rth3url8n7gvucdtobybdq5h"	true	"SporkBot"	false	false		"page, page"
+820078733	2018-01-12 23:28:39	56237369	"User:Editingaccount1994/sandbox"	2	false		false	12406635	"Replace template per [[Wikipedia:Templates for discussion/Log/2011 February 17|TFD outcome]]; no change in content"	27782		"531dizmmloyxffbkdr5vph7owh921eg"	true	"SporkBot"	false	false		"page, page"
+820177382	2018-01-13 13:45:33	56237369	"User:Editingaccount1994/sandbox"	2	false		false	13791031	"translate TestCaseD if you are from tor you need neutral point of view "	27757		"nik9p2u2fuk4yazjxt8ymbicxv5qid9"	false	"Frietjes"	false	false		"page, page"
+822038928	2018-01-24 01:35:22	56237369	"User:Editingaccount1994/sandbox"	2	false		false	2304267	"Removing [[:c:File:Li_Chevalier_Art_Studio.jpg|Li_Chevalier_Art_Studio.jpg]], it has been deleted from Commons by [[:c:User:JuTa|JuTa]] because: [[:c:COM:OTRS|No permission]] since 16 January 2018."	27667		"gwk6pampl8si1v5pv3kwgteg710sfw3"	false	"CommonsDelinker"	false	false		"page, page"
+819091874	2018-01-07 10:42:20	56237370	"Anita del Rey"	0	true	"Ana del Rey"	false	1368779	"r from alt name"	25		"n4ozbsgle13p9yywtfrz982ccj8woc9"	false	"PamD"	false	false		
+819091883	2018-01-07 10:42:27	56237371	"User talk:119.94.96.157"	3	false		false	13286072	"Warning [[Special:Contributions/119.94.96.157|119.94.96.157]] - #1"	1274		"ksohnvsbeuzwpl5vb8a3v8m18hva0a7"	false	"ClueBot NG"	false	false		"page"
+819091914	2018-01-07 10:42:50	56237372	"Category:Ohmi Railway"	14	false		false	677153	"[[WP:AES|←]]Created page with ' [[Category:Railway companies of Japan]] [[Category:Rail transport in Shiga Prefecture]] [[Category:Seibu Group]]'"	113		"je7aw21fedbwyqsyofpisdrynsu7olr"	false	"Underbar dk"	false	false		
+819091968	2018-01-07 10:43:32	56237375	"User talk:92.226.219.222"	3	false		false	882433	"[[WP:AES|←]]Created page with '{{3rr}}~~~~'"	199		"cpm4tkzcx4hc6irr9ukbi06ogud8dtq"	false	"TastyPoutine"	false	false		
+819094036	2018-01-07 11:10:24	56237375	"User talk:92.226.219.222"	3	false		false	7611264	"[[User:AnomieBOT/docs/TemplateSubster|Substing templates]]: {{3rr}}. See [[User:AnomieBOT/docs/TemplateSubster]] for info."	1840		"artmfz8b2gxhb3pp8a5p4ksplxqfkpg"	true	"AnomieBOT"	false	false		"page, page, page, page"
+819112363	2018-01-07 14:33:36	56237375	"User talk:92.226.219.222"	3	false		false	702940	"Your IP address has been blocked from editing because it has been used to [[WP:EVADE|evade a previous block]]. ([[WP:TW|TW]])"	2949		"dn9wj0n8d8pdd5lqe56uw5xamupowr1"	false	"Only"	false	false		"page, page, page, page, page, page"
+819092004	2018-01-07 10:44:01	56237376	"User:Dipayanacharya"	2	false		false	32794237	"Education"	28		"ofueugwatmmn7u73isw732neuza57gk"	false	"Dipayanacharya"	false	false		
+819092390	2018-01-07 10:49:08	56237376	"User:Dipayanacharya"	2	false		false	32794237	"School"	38		"dsz55xv96ec2uv6w9c1z7c52ipfovbw"	false	"Dipayanacharya"	false	false		
+819092066	2018-01-07 10:44:56	56237378	"BSCIC"	0	true	"Bangladesh Small and Cottage Industries Corporation"	false	21516552	"[[WP:AES|←]]Redirected page to [[Bangladesh Small and Cottage Industries Corporation]]"	65		"9ma38hak0ef1ew4fpiutxpnzd8oz1wd"	false	"Vinegarymass911"	false	false		
+819092102	2018-01-07 10:45:21	56237379	"Category:Women government ministers of Yemen"	14	false		false	754619	"[[WP:AES|←]]Created page with '{{portal|Yemen|Politics}} {{Non-diffusing subcategory|Government ministers of Yemen}}  {{Underpopulated category}}  Category:Women government ministers by nati...'"	285		"4dvakoat58bzyf5hmtthxukt29hip6n"	false	"BrownHairedGirl"	false	false		
+819092135	2018-01-07 10:45:54	56237381	"Talk:List of Morning Glories Characters"	1	false		false	410898	"[[WP:AES|←]]Created page with '{{WikiProject Fictional characters|class=List|importance=low}} {{Comicsproj|class=List|importance=low}}'"	103		"2sjrxsc7os9k9pg4su2t4rk2j8nn0h7"	false	"PRehse"	false	false		
+819092138	2018-01-07 10:45:56	56237382	"User talk:106.207.126.114"	3	false		false	13286072	"Warning [[Special:Contributions/106.207.126.114|106.207.126.114]] - #1"	1330		"3y9t5wpk6ur5jhone75rhm4wjf01fgi"	false	"ClueBot NG"	false	false		"page"
+819092495	2018-01-07 10:50:22	56237382	"User talk:106.207.126.114"	3	false		false	31190506	"Caution: Unconstructive editing on [[List of Baahubali characters]]. ([[WP:TW|TW]])"	2355		"8wvn6vh3isyt0dorpe89lztrburgupe"	false	"HindWIKI"	false	false		"page"

test/baseline_output/basic_regextest_2.tsv

@@ -1,27 +1,27 @@
-"revid"	"date_time"	"articleid"	"title"	"namespace"	"deleted"	"editorid"	"edit_summary"	"text_chars"	"reverteds"	"sha1"	"minor"	"editor"	"anon"	"revert"	"wiki_welcome"	"chev_com"	"warning"
-819091731	2018-01-07 10:40:58	56237363	"User talk:86.139.142.254"	3	false	3742946	"Your IP address has been blocked from editing because it has been used to [[WP:EVADE|evade a previous block]]. ([[WP:TW|TW]])"	1141		"135nz8q6lfam6cojla7azb7k5alx3t3"	false	"NinjaRobotPirate"	false	false			
-819091755	2018-01-07 10:41:10	56237364	"User talk:Kavin kavitha"	3	false	32792125	"[[WP:AES|←]]Created page with ''''''Kavin (Tamil. கவின்) is a masculine given name, which is Tamil for ""beauty"", ""grace"", ""fairness"" or ""comeliness""Kavin is born on 01 /12/2001 at Sa...'"	663		"0pwezjc6yopz0smc8al6ogc4fax5bwo"	false	"Kavin kavitha"	false	false			
-819091788	2018-01-07 10:41:26	56237365	"User talk:Dr.vivek163"	3	false	32621254	"/* Regarding Merger discussion */ new section"	399		"sz3t2ap7z8bpkdvdvi195f3i35949bv"	false	"Amicable always"	false	false			
-819091796	2018-01-07 10:41:31	56237366	"User talk:Twistorl"	3	false	13286072	"Warning [[Special:Contributions/Twistorl|Twistorl]] - #1"	1260		"r6s5j8j3iykenrhuhpnkpsmmd71vubf"	false	"ClueBot NG"	false	false	"welcome to Wikipedia"		"Warning"
-819091825	2018-01-07 10:41:51	56237368	"Kom Firin"	0	false	8409334	"[[WP:AES|←]]Created page with '[[File:Stele 67.119 Brooklyn.jpg|thumb|Stele of the [[Libu#Great Chiefs of the Libu|Chief of the Libu]] Titaru, a contemporary of pharaoh [[Shoshenq V]] of the [...'TestCaseB and you're a Tor node "	2249		"tf5qz2yaswx61zrlm9ovxzuhl7r2dc4"	false	"Khruner"	false	false			
-822610647	2018-01-27 12:16:02	56237368	"Kom Firin"	0	false	8409334	"/* History */ typo"	2230		"e6oa4g0qv64icdaq26uu1zzbyr5hcbh"	true	"Khruner"	false	false			
-819091844	2018-01-07 10:42:05	56237369	"User:Editingaccount1994/sandbox"	2	false	32794215	"[[WP:AES|←]]Created page with '{{User sandbox}} <!-- EDIT BELOW THIS LINE --> {{voir homonymes|Chevalier}} {{Infobox Artiste  | nom                = Li Chevalier  | autres noms        =   | im...'"	27840		"0fyvyh2a8xu41gt8obr34oba0bfixj6"	false	"Editingaccount1994"	false	false		"Chevalier, Chevalier"	
-819093984	2018-01-07 11:09:52	56237369	"User:Editingaccount1994/sandbox"	2	false	7611264	"[[User:AnomieBOT/docs/TemplateSubster|Substing templates]]: {{Lien web}}. See [[User:AnomieBOT/docs/TemplateSubster]] for info."	27787		"8gy52aolt5rg3eaketwj5v7eiw0apv2"	true	"AnomieBOT"	false	false			
-820064189	2018-01-12 21:45:50	56237369	"User:Editingaccount1994/sandbox"	2	false	12406635	"Orphan per [[WP:TFD|TFD outcome]]"	27784		"he8ydemaanxlrpftqxkez8jfpge1fsj"	true	"SporkBot"	false	false			
-820078679	2018-01-12 23:28:11	56237369	"User:Editingaccount1994/sandbox"	2	false	12406635	"Replace template per [[Wikipedia:Templates for discussion/Log/2010 June 13|TFD outcome]]; no change in content"	27783		"0to17w9rth3url8n7gvucdtobybdq5h"	true	"SporkBot"	false	false			
-820078733	2018-01-12 23:28:39	56237369	"User:Editingaccount1994/sandbox"	2	false	12406635	"Replace template per [[Wikipedia:Templates for discussion/Log/2011 February 17|TFD outcome]]; no change in content"	27782		"531dizmmloyxffbkdr5vph7owh921eg"	true	"SporkBot"	false	false			
-820177382	2018-01-13 13:45:33	56237369	"User:Editingaccount1994/sandbox"	2	false	13791031	"translate TestCaseD if you are from tor you need neutral point of view "	27757		"nik9p2u2fuk4yazjxt8ymbicxv5qid9"	false	"Frietjes"	false	false			
-822038928	2018-01-24 01:35:22	56237369	"User:Editingaccount1994/sandbox"	2	false	2304267	"Removing [[:c:File:Li_Chevalier_Art_Studio.jpg|Li_Chevalier_Art_Studio.jpg]], it has been deleted from Commons by [[:c:User:JuTa|JuTa]] because: [[:c:COM:OTRS|No permission]] since 16 January 2018."	27667		"gwk6pampl8si1v5pv3kwgteg710sfw3"	false	"CommonsDelinker"	false	false		"Chevalier, Chevalier"	
-819091874	2018-01-07 10:42:20	56237370	"Anita del Rey"	0	false	1368779	"r from alt name"	25		"n4ozbsgle13p9yywtfrz982ccj8woc9"	false	"PamD"	false	false			
-819091883	2018-01-07 10:42:27	56237371	"User talk:119.94.96.157"	3	false	13286072	"Warning [[Special:Contributions/119.94.96.157|119.94.96.157]] - #1"	1274		"ksohnvsbeuzwpl5vb8a3v8m18hva0a7"	false	"ClueBot NG"	false	false	"welcome to Wikipedia"		"Warning"
-819091914	2018-01-07 10:42:50	56237372	"Category:Ohmi Railway"	14	false	677153	"[[WP:AES|←]]Created page with ' [[Category:Railway companies of Japan]] [[Category:Rail transport in Shiga Prefecture]] [[Category:Seibu Group]]'"	113		"je7aw21fedbwyqsyofpisdrynsu7olr"	false	"Underbar dk"	false	false			
-819091968	2018-01-07 10:43:32	56237375	"User talk:92.226.219.222"	3	false	882433	"[[WP:AES|←]]Created page with '{{3rr}}~~~~'"	199		"cpm4tkzcx4hc6irr9ukbi06ogud8dtq"	false	"TastyPoutine"	false	false			
-819094036	2018-01-07 11:10:24	56237375	"User talk:92.226.219.222"	3	false	7611264	"[[User:AnomieBOT/docs/TemplateSubster|Substing templates]]: {{3rr}}. See [[User:AnomieBOT/docs/TemplateSubster]] for info."	1840		"artmfz8b2gxhb3pp8a5p4ksplxqfkpg"	true	"AnomieBOT"	false	false			
-819112363	2018-01-07 14:33:36	56237375	"User talk:92.226.219.222"	3	false	702940	"Your IP address has been blocked from editing because it has been used to [[WP:EVADE|evade a previous block]]. ([[WP:TW|TW]])"	2949		"dn9wj0n8d8pdd5lqe56uw5xamupowr1"	false	"Only"	false	false			
-819092004	2018-01-07 10:44:01	56237376	"User:Dipayanacharya"	2	false	32794237	"Education"	28		"ofueugwatmmn7u73isw732neuza57gk"	false	"Dipayanacharya"	false	false			
-819092390	2018-01-07 10:49:08	56237376	"User:Dipayanacharya"	2	false	32794237	"School"	38		"dsz55xv96ec2uv6w9c1z7c52ipfovbw"	false	"Dipayanacharya"	false	false			
-819092066	2018-01-07 10:44:56	56237378	"BSCIC"	0	false	21516552	"[[WP:AES|←]]Redirected page to [[Bangladesh Small and Cottage Industries Corporation]]"	65		"9ma38hak0ef1ew4fpiutxpnzd8oz1wd"	false	"Vinegarymass911"	false	false			
-819092102	2018-01-07 10:45:21	56237379	"Category:Women government ministers of Yemen"	14	false	754619	"[[WP:AES|←]]Created page with '{{portal|Yemen|Politics}} {{Non-diffusing subcategory|Government ministers of Yemen}}  {{Underpopulated category}}  Category:Women government ministers by nati...'"	285		"4dvakoat58bzyf5hmtthxukt29hip6n"	false	"BrownHairedGirl"	false	false			
-819092135	2018-01-07 10:45:54	56237381	"Talk:List of Morning Glories Characters"	1	false	410898	"[[WP:AES|←]]Created page with '{{WikiProject Fictional characters|class=List|importance=low}} {{Comicsproj|class=List|importance=low}}'"	103		"2sjrxsc7os9k9pg4su2t4rk2j8nn0h7"	false	"PRehse"	false	false			
-819092138	2018-01-07 10:45:56	56237382	"User talk:106.207.126.114"	3	false	13286072	"Warning [[Special:Contributions/106.207.126.114|106.207.126.114]] - #1"	1330		"3y9t5wpk6ur5jhone75rhm4wjf01fgi"	false	"ClueBot NG"	false	false	"welcome to Wikipedia"		"Warning"
-819092495	2018-01-07 10:50:22	56237382	"User talk:106.207.126.114"	3	false	31190506	"Caution: Unconstructive editing on [[List of Baahubali characters]]. ([[WP:TW|TW]])"	2355		"8wvn6vh3isyt0dorpe89lztrburgupe"	false	"HindWIKI"	false	false	"welcome to Wikipedia"		
+"revid"	"date_time"	"articleid"	"title"	"namespace"	"revision_is_redirect"	"revision_redirect_target"	"deleted"	"editorid"	"edit_summary"	"text_chars"	"reverteds"	"sha1"	"minor"	"editor"	"anon"	"revert"	"wiki_welcome"	"chev_com"	"warning"
+819091731	2018-01-07 10:40:58	56237363	"User talk:86.139.142.254"	3	false		false	3742946	"Your IP address has been blocked from editing because it has been used to [[WP:EVADE|evade a previous block]]. ([[WP:TW|TW]])"	1141		"135nz8q6lfam6cojla7azb7k5alx3t3"	false	"NinjaRobotPirate"	false	false			
+819091755	2018-01-07 10:41:10	56237364	"User talk:Kavin kavitha"	3	false		false	32792125	"[[WP:AES|←]]Created page with ''''''Kavin (Tamil. கவின்) is a masculine given name, which is Tamil for ""beauty"", ""grace"", ""fairness"" or ""comeliness""Kavin is born on 01 /12/2001 at Sa...'"	663		"0pwezjc6yopz0smc8al6ogc4fax5bwo"	false	"Kavin kavitha"	false	false			
+819091788	2018-01-07 10:41:26	56237365	"User talk:Dr.vivek163"	3	false		false	32621254	"/* Regarding Merger discussion */ new section"	399		"sz3t2ap7z8bpkdvdvi195f3i35949bv"	false	"Amicable always"	false	false			
+819091796	2018-01-07 10:41:31	56237366	"User talk:Twistorl"	3	false		false	13286072	"Warning [[Special:Contributions/Twistorl|Twistorl]] - #1"	1260		"r6s5j8j3iykenrhuhpnkpsmmd71vubf"	false	"ClueBot NG"	false	false	"welcome to Wikipedia"		"Warning"
+819091825	2018-01-07 10:41:51	56237368	"Kom Firin"	0	false		false	8409334	"[[WP:AES|←]]Created page with '[[File:Stele 67.119 Brooklyn.jpg|thumb|Stele of the [[Libu#Great Chiefs of the Libu|Chief of the Libu]] Titaru, a contemporary of pharaoh [[Shoshenq V]] of the [...'TestCaseB and you're a Tor node "	2249		"tf5qz2yaswx61zrlm9ovxzuhl7r2dc4"	false	"Khruner"	false	false			
+822610647	2018-01-27 12:16:02	56237368	"Kom Firin"	0	false		false	8409334	"/* History */ typo"	2230		"e6oa4g0qv64icdaq26uu1zzbyr5hcbh"	true	"Khruner"	false	false			
+819091844	2018-01-07 10:42:05	56237369	"User:Editingaccount1994/sandbox"	2	false		false	32794215	"[[WP:AES|←]]Created page with '{{User sandbox}} <!-- EDIT BELOW THIS LINE --> {{voir homonymes|Chevalier}} {{Infobox Artiste  | nom                = Li Chevalier  | autres noms        =   | im...'"	27840		"0fyvyh2a8xu41gt8obr34oba0bfixj6"	false	"Editingaccount1994"	false	false		"Chevalier, Chevalier"	
+819093984	2018-01-07 11:09:52	56237369	"User:Editingaccount1994/sandbox"	2	false		false	7611264	"[[User:AnomieBOT/docs/TemplateSubster|Substing templates]]: {{Lien web}}. See [[User:AnomieBOT/docs/TemplateSubster]] for info."	27787		"8gy52aolt5rg3eaketwj5v7eiw0apv2"	true	"AnomieBOT"	false	false			
+820064189	2018-01-12 21:45:50	56237369	"User:Editingaccount1994/sandbox"	2	false		false	12406635	"Orphan per [[WP:TFD|TFD outcome]]"	27784		"he8ydemaanxlrpftqxkez8jfpge1fsj"	true	"SporkBot"	false	false			
+820078679	2018-01-12 23:28:11	56237369	"User:Editingaccount1994/sandbox"	2	false		false	12406635	"Replace template per [[Wikipedia:Templates for discussion/Log/2010 June 13|TFD outcome]]; no change in content"	27783		"0to17w9rth3url8n7gvucdtobybdq5h"	true	"SporkBot"	false	false			
+820078733	2018-01-12 23:28:39	56237369	"User:Editingaccount1994/sandbox"	2	false		false	12406635	"Replace template per [[Wikipedia:Templates for discussion/Log/2011 February 17|TFD outcome]]; no change in content"	27782		"531dizmmloyxffbkdr5vph7owh921eg"	true	"SporkBot"	false	false			
+820177382	2018-01-13 13:45:33	56237369	"User:Editingaccount1994/sandbox"	2	false		false	13791031	"translate TestCaseD if you are from tor you need neutral point of view "	27757		"nik9p2u2fuk4yazjxt8ymbicxv5qid9"	false	"Frietjes"	false	false			
+822038928	2018-01-24 01:35:22	56237369	"User:Editingaccount1994/sandbox"	2	false		false	2304267	"Removing [[:c:File:Li_Chevalier_Art_Studio.jpg|Li_Chevalier_Art_Studio.jpg]], it has been deleted from Commons by [[:c:User:JuTa|JuTa]] because: [[:c:COM:OTRS|No permission]] since 16 January 2018."	27667		"gwk6pampl8si1v5pv3kwgteg710sfw3"	false	"CommonsDelinker"	false	false		"Chevalier, Chevalier"	
+819091874	2018-01-07 10:42:20	56237370	"Anita del Rey"	0	true	"Ana del Rey"	false	1368779	"r from alt name"	25		"n4ozbsgle13p9yywtfrz982ccj8woc9"	false	"PamD"	false	false			
+819091883	2018-01-07 10:42:27	56237371	"User talk:119.94.96.157"	3	false		false	13286072	"Warning [[Special:Contributions/119.94.96.157|119.94.96.157]] - #1"	1274		"ksohnvsbeuzwpl5vb8a3v8m18hva0a7"	false	"ClueBot NG"	false	false	"welcome to Wikipedia"		"Warning"
+819091914	2018-01-07 10:42:50	56237372	"Category:Ohmi Railway"	14	false		false	677153	"[[WP:AES|←]]Created page with ' [[Category:Railway companies of Japan]] [[Category:Rail transport in Shiga Prefecture]] [[Category:Seibu Group]]'"	113		"je7aw21fedbwyqsyofpisdrynsu7olr"	false	"Underbar dk"	false	false			
+819091968	2018-01-07 10:43:32	56237375	"User talk:92.226.219.222"	3	false		false	882433	"[[WP:AES|←]]Created page with '{{3rr}}~~~~'"	199		"cpm4tkzcx4hc6irr9ukbi06ogud8dtq"	false	"TastyPoutine"	false	false			
+819094036	2018-01-07 11:10:24	56237375	"User talk:92.226.219.222"	3	false		false	7611264	"[[User:AnomieBOT/docs/TemplateSubster|Substing templates]]: {{3rr}}. See [[User:AnomieBOT/docs/TemplateSubster]] for info."	1840		"artmfz8b2gxhb3pp8a5p4ksplxqfkpg"	true	"AnomieBOT"	false	false			
+819112363	2018-01-07 14:33:36	56237375	"User talk:92.226.219.222"	3	false		false	702940	"Your IP address has been blocked from editing because it has been used to [[WP:EVADE|evade a previous block]]. ([[WP:TW|TW]])"	2949		"dn9wj0n8d8pdd5lqe56uw5xamupowr1"	false	"Only"	false	false			
+819092004	2018-01-07 10:44:01	56237376	"User:Dipayanacharya"	2	false		false	32794237	"Education"	28		"ofueugwatmmn7u73isw732neuza57gk"	false	"Dipayanacharya"	false	false			
+819092390	2018-01-07 10:49:08	56237376	"User:Dipayanacharya"	2	false		false	32794237	"School"	38		"dsz55xv96ec2uv6w9c1z7c52ipfovbw"	false	"Dipayanacharya"	false	false			
+819092066	2018-01-07 10:44:56	56237378	"BSCIC"	0	true	"Bangladesh Small and Cottage Industries Corporation"	false	21516552	"[[WP:AES|←]]Redirected page to [[Bangladesh Small and Cottage Industries Corporation]]"	65		"9ma38hak0ef1ew4fpiutxpnzd8oz1wd"	false	"Vinegarymass911"	false	false			
+819092102	2018-01-07 10:45:21	56237379	"Category:Women government ministers of Yemen"	14	false		false	754619	"[[WP:AES|←]]Created page with '{{portal|Yemen|Politics}} {{Non-diffusing subcategory|Government ministers of Yemen}}  {{Underpopulated category}}  Category:Women government ministers by nati...'"	285		"4dvakoat58bzyf5hmtthxukt29hip6n"	false	"BrownHairedGirl"	false	false			
+819092135	2018-01-07 10:45:54	56237381	"Talk:List of Morning Glories Characters"	1	false		false	410898	"[[WP:AES|←]]Created page with '{{WikiProject Fictional characters|class=List|importance=low}} {{Comicsproj|class=List|importance=low}}'"	103		"2sjrxsc7os9k9pg4su2t4rk2j8nn0h7"	false	"PRehse"	false	false			
+819092138	2018-01-07 10:45:56	56237382	"User talk:106.207.126.114"	3	false		false	13286072	"Warning [[Special:Contributions/106.207.126.114|106.207.126.114]] - #1"	1330		"3y9t5wpk6ur5jhone75rhm4wjf01fgi"	false	"ClueBot NG"	false	false	"welcome to Wikipedia"		"Warning"
+819092495	2018-01-07 10:50:22	56237382	"User talk:106.207.126.114"	3	false		false	31190506	"Caution: Unconstructive editing on [[List of Baahubali characters]]. ([[WP:TW|TW]])"	2355		"8wvn6vh3isyt0dorpe89lztrburgupe"	false	"HindWIKI"	false	false	"welcome to Wikipedia"		

test/baseline_output/basic_regextest_3.tsv

@@ -1,27 +1,27 @@
-"revid"	"date_time"	"articleid"	"title"	"namespace"	"deleted"	"editorid"	"edit_summary"	"text_chars"	"reverteds"	"sha1"	"minor"	"editor"	"anon"	"revert"	"wp_evade"
-819091731	2018-01-07 10:40:58	56237363	"User talk:86.139.142.254"	3	false	3742946	"Your IP address has been blocked from editing because it has been used to [[WP:EVADE|evade a previous block]]. ([[WP:TW|TW]])"	1141		"135nz8q6lfam6cojla7azb7k5alx3t3"	false	"NinjaRobotPirate"	false	false	"WP:EVADE"
-819091755	2018-01-07 10:41:10	56237364	"User talk:Kavin kavitha"	3	false	32792125	"[[WP:AES|←]]Created page with ''''''Kavin (Tamil. கவின்) is a masculine given name, which is Tamil for ""beauty"", ""grace"", ""fairness"" or ""comeliness""Kavin is born on 01 /12/2001 at Sa...'"	663		"0pwezjc6yopz0smc8al6ogc4fax5bwo"	false	"Kavin kavitha"	false	false	
-819091788	2018-01-07 10:41:26	56237365	"User talk:Dr.vivek163"	3	false	32621254	"/* Regarding Merger discussion */ new section"	399		"sz3t2ap7z8bpkdvdvi195f3i35949bv"	false	"Amicable always"	false	false	
-819091796	2018-01-07 10:41:31	56237366	"User talk:Twistorl"	3	false	13286072	"Warning [[Special:Contributions/Twistorl|Twistorl]] - #1"	1260		"r6s5j8j3iykenrhuhpnkpsmmd71vubf"	false	"ClueBot NG"	false	false	
-819091825	2018-01-07 10:41:51	56237368	"Kom Firin"	0	false	8409334	"[[WP:AES|←]]Created page with '[[File:Stele 67.119 Brooklyn.jpg|thumb|Stele of the [[Libu#Great Chiefs of the Libu|Chief of the Libu]] Titaru, a contemporary of pharaoh [[Shoshenq V]] of the [...'TestCaseB and you're a Tor node "	2249		"tf5qz2yaswx61zrlm9ovxzuhl7r2dc4"	false	"Khruner"	false	false	
-822610647	2018-01-27 12:16:02	56237368	"Kom Firin"	0	false	8409334	"/* History */ typo"	2230		"e6oa4g0qv64icdaq26uu1zzbyr5hcbh"	true	"Khruner"	false	false	
-819091844	2018-01-07 10:42:05	56237369	"User:Editingaccount1994/sandbox"	2	false	32794215	"[[WP:AES|←]]Created page with '{{User sandbox}} <!-- EDIT BELOW THIS LINE --> {{voir homonymes|Chevalier}} {{Infobox Artiste  | nom                = Li Chevalier  | autres noms        =   | im...'"	27840		"0fyvyh2a8xu41gt8obr34oba0bfixj6"	false	"Editingaccount1994"	false	false	
-819093984	2018-01-07 11:09:52	56237369	"User:Editingaccount1994/sandbox"	2	false	7611264	"[[User:AnomieBOT/docs/TemplateSubster|Substing templates]]: {{Lien web}}. See [[User:AnomieBOT/docs/TemplateSubster]] for info."	27787		"8gy52aolt5rg3eaketwj5v7eiw0apv2"	true	"AnomieBOT"	false	false	
-820064189	2018-01-12 21:45:50	56237369	"User:Editingaccount1994/sandbox"	2	false	12406635	"Orphan per [[WP:TFD|TFD outcome]]"	27784		"he8ydemaanxlrpftqxkez8jfpge1fsj"	true	"SporkBot"	false	false	
-820078679	2018-01-12 23:28:11	56237369	"User:Editingaccount1994/sandbox"	2	false	12406635	"Replace template per [[Wikipedia:Templates for discussion/Log/2010 June 13|TFD outcome]]; no change in content"	27783		"0to17w9rth3url8n7gvucdtobybdq5h"	true	"SporkBot"	false	false	
-820078733	2018-01-12 23:28:39	56237369	"User:Editingaccount1994/sandbox"	2	false	12406635	"Replace template per [[Wikipedia:Templates for discussion/Log/2011 February 17|TFD outcome]]; no change in content"	27782		"531dizmmloyxffbkdr5vph7owh921eg"	true	"SporkBot"	false	false	
-820177382	2018-01-13 13:45:33	56237369	"User:Editingaccount1994/sandbox"	2	false	13791031	"translate TestCaseD if you are from tor you need neutral point of view "	27757		"nik9p2u2fuk4yazjxt8ymbicxv5qid9"	false	"Frietjes"	false	false	
-822038928	2018-01-24 01:35:22	56237369	"User:Editingaccount1994/sandbox"	2	false	2304267	"Removing [[:c:File:Li_Chevalier_Art_Studio.jpg|Li_Chevalier_Art_Studio.jpg]], it has been deleted from Commons by [[:c:User:JuTa|JuTa]] because: [[:c:COM:OTRS|No permission]] since 16 January 2018."	27667		"gwk6pampl8si1v5pv3kwgteg710sfw3"	false	"CommonsDelinker"	false	false	
-819091874	2018-01-07 10:42:20	56237370	"Anita del Rey"	0	false	1368779	"r from alt name"	25		"n4ozbsgle13p9yywtfrz982ccj8woc9"	false	"PamD"	false	false	
-819091883	2018-01-07 10:42:27	56237371	"User talk:119.94.96.157"	3	false	13286072	"Warning [[Special:Contributions/119.94.96.157|119.94.96.157]] - #1"	1274		"ksohnvsbeuzwpl5vb8a3v8m18hva0a7"	false	"ClueBot NG"	false	false	
-819091914	2018-01-07 10:42:50	56237372	"Category:Ohmi Railway"	14	false	677153	"[[WP:AES|←]]Created page with ' [[Category:Railway companies of Japan]] [[Category:Rail transport in Shiga Prefecture]] [[Category:Seibu Group]]'"	113		"je7aw21fedbwyqsyofpisdrynsu7olr"	false	"Underbar dk"	false	false	
-819091968	2018-01-07 10:43:32	56237375	"User talk:92.226.219.222"	3	false	882433	"[[WP:AES|←]]Created page with '{{3rr}}~~~~'"	199		"cpm4tkzcx4hc6irr9ukbi06ogud8dtq"	false	"TastyPoutine"	false	false	
-819094036	2018-01-07 11:10:24	56237375	"User talk:92.226.219.222"	3	false	7611264	"[[User:AnomieBOT/docs/TemplateSubster|Substing templates]]: {{3rr}}. See [[User:AnomieBOT/docs/TemplateSubster]] for info."	1840		"artmfz8b2gxhb3pp8a5p4ksplxqfkpg"	true	"AnomieBOT"	false	false	
-819112363	2018-01-07 14:33:36	56237375	"User talk:92.226.219.222"	3	false	702940	"Your IP address has been blocked from editing because it has been used to [[WP:EVADE|evade a previous block]]. ([[WP:TW|TW]])"	2949		"dn9wj0n8d8pdd5lqe56uw5xamupowr1"	false	"Only"	false	false	"WP:EVADE"
-819092004	2018-01-07 10:44:01	56237376	"User:Dipayanacharya"	2	false	32794237	"Education"	28		"ofueugwatmmn7u73isw732neuza57gk"	false	"Dipayanacharya"	false	false	
-819092390	2018-01-07 10:49:08	56237376	"User:Dipayanacharya"	2	false	32794237	"School"	38		"dsz55xv96ec2uv6w9c1z7c52ipfovbw"	false	"Dipayanacharya"	false	false	
-819092066	2018-01-07 10:44:56	56237378	"BSCIC"	0	false	21516552	"[[WP:AES|←]]Redirected page to [[Bangladesh Small and Cottage Industries Corporation]]"	65		"9ma38hak0ef1ew4fpiutxpnzd8oz1wd"	false	"Vinegarymass911"	false	false	
-819092102	2018-01-07 10:45:21	56237379	"Category:Women government ministers of Yemen"	14	false	754619	"[[WP:AES|←]]Created page with '{{portal|Yemen|Politics}} {{Non-diffusing subcategory|Government ministers of Yemen}}  {{Underpopulated category}}  Category:Women government ministers by nati...'"	285		"4dvakoat58bzyf5hmtthxukt29hip6n"	false	"BrownHairedGirl"	false	false	
-819092135	2018-01-07 10:45:54	56237381	"Talk:List of Morning Glories Characters"	1	false	410898	"[[WP:AES|←]]Created page with '{{WikiProject Fictional characters|class=List|importance=low}} {{Comicsproj|class=List|importance=low}}'"	103		"2sjrxsc7os9k9pg4su2t4rk2j8nn0h7"	false	"PRehse"	false	false	
-819092138	2018-01-07 10:45:56	56237382	"User talk:106.207.126.114"	3	false	13286072	"Warning [[Special:Contributions/106.207.126.114|106.207.126.114]] - #1"	1330		"3y9t5wpk6ur5jhone75rhm4wjf01fgi"	false	"ClueBot NG"	false	false	
-819092495	2018-01-07 10:50:22	56237382	"User talk:106.207.126.114"	3	false	31190506	"Caution: Unconstructive editing on [[List of Baahubali characters]]. ([[WP:TW|TW]])"	2355		"8wvn6vh3isyt0dorpe89lztrburgupe"	false	"HindWIKI"	false	false	
+"revid"	"date_time"	"articleid"	"title"	"namespace"	"revision_is_redirect"	"revision_redirect_target"	"deleted"	"editorid"	"edit_summary"	"text_chars"	"reverteds"	"sha1"	"minor"	"editor"	"anon"	"revert"	"wp_evade"
+819091731	2018-01-07 10:40:58	56237363	"User talk:86.139.142.254"	3	false		false	3742946	"Your IP address has been blocked from editing because it has been used to [[WP:EVADE|evade a previous block]]. ([[WP:TW|TW]])"	1141		"135nz8q6lfam6cojla7azb7k5alx3t3"	false	"NinjaRobotPirate"	false	false	"WP:EVADE"
+819091755	2018-01-07 10:41:10	56237364	"User talk:Kavin kavitha"	3	false		false	32792125	"[[WP:AES|←]]Created page with ''''''Kavin (Tamil. கவின்) is a masculine given name, which is Tamil for ""beauty"", ""grace"", ""fairness"" or ""comeliness""Kavin is born on 01 /12/2001 at Sa...'"	663		"0pwezjc6yopz0smc8al6ogc4fax5bwo"	false	"Kavin kavitha"	false	false	
+819091788	2018-01-07 10:41:26	56237365	"User talk:Dr.vivek163"	3	false		false	32621254	"/* Regarding Merger discussion */ new section"	399		"sz3t2ap7z8bpkdvdvi195f3i35949bv"	false	"Amicable always"	false	false	
+819091796	2018-01-07 10:41:31	56237366	"User talk:Twistorl"	3	false		false	13286072	"Warning [[Special:Contributions/Twistorl|Twistorl]] - #1"	1260		"r6s5j8j3iykenrhuhpnkpsmmd71vubf"	false	"ClueBot NG"	false	false	
+819091825	2018-01-07 10:41:51	56237368	"Kom Firin"	0	false		false	8409334	"[[WP:AES|←]]Created page with '[[File:Stele 67.119 Brooklyn.jpg|thumb|Stele of the [[Libu#Great Chiefs of the Libu|Chief of the Libu]] Titaru, a contemporary of pharaoh [[Shoshenq V]] of the [...'TestCaseB and you're a Tor node "	2249		"tf5qz2yaswx61zrlm9ovxzuhl7r2dc4"	false	"Khruner"	false	false	
+822610647	2018-01-27 12:16:02	56237368	"Kom Firin"	0	false		false	8409334	"/* History */ typo"	2230		"e6oa4g0qv64icdaq26uu1zzbyr5hcbh"	true	"Khruner"	false	false	
+819091844	2018-01-07 10:42:05	56237369	"User:Editingaccount1994/sandbox"	2	false		false	32794215	"[[WP:AES|←]]Created page with '{{User sandbox}} <!-- EDIT BELOW THIS LINE --> {{voir homonymes|Chevalier}} {{Infobox Artiste  | nom                = Li Chevalier  | autres noms        =   | im...'"	27840		"0fyvyh2a8xu41gt8obr34oba0bfixj6"	false	"Editingaccount1994"	false	false	
+819093984	2018-01-07 11:09:52	56237369	"User:Editingaccount1994/sandbox"	2	false		false	7611264	"[[User:AnomieBOT/docs/TemplateSubster|Substing templates]]: {{Lien web}}. See [[User:AnomieBOT/docs/TemplateSubster]] for info."	27787		"8gy52aolt5rg3eaketwj5v7eiw0apv2"	true	"AnomieBOT"	false	false	
+820064189	2018-01-12 21:45:50	56237369	"User:Editingaccount1994/sandbox"	2	false		false	12406635	"Orphan per [[WP:TFD|TFD outcome]]"	27784		"he8ydemaanxlrpftqxkez8jfpge1fsj"	true	"SporkBot"	false	false	
+820078679	2018-01-12 23:28:11	56237369	"User:Editingaccount1994/sandbox"	2	false		false	12406635	"Replace template per [[Wikipedia:Templates for discussion/Log/2010 June 13|TFD outcome]]; no change in content"	27783		"0to17w9rth3url8n7gvucdtobybdq5h"	true	"SporkBot"	false	false	
+820078733	2018-01-12 23:28:39	56237369	"User:Editingaccount1994/sandbox"	2	false		false	12406635	"Replace template per [[Wikipedia:Templates for discussion/Log/2011 February 17|TFD outcome]]; no change in content"	27782		"531dizmmloyxffbkdr5vph7owh921eg"	true	"SporkBot"	false	false	
+820177382	2018-01-13 13:45:33	56237369	"User:Editingaccount1994/sandbox"	2	false		false	13791031	"translate TestCaseD if you are from tor you need neutral point of view "	27757		"nik9p2u2fuk4yazjxt8ymbicxv5qid9"	false	"Frietjes"	false	false	
+822038928	2018-01-24 01:35:22	56237369	"User:Editingaccount1994/sandbox"	2	false		false	2304267	"Removing [[:c:File:Li_Chevalier_Art_Studio.jpg|Li_Chevalier_Art_Studio.jpg]], it has been deleted from Commons by [[:c:User:JuTa|JuTa]] because: [[:c:COM:OTRS|No permission]] since 16 January 2018."	27667		"gwk6pampl8si1v5pv3kwgteg710sfw3"	false	"CommonsDelinker"	false	false	
+819091874	2018-01-07 10:42:20	56237370	"Anita del Rey"	0	true	"Ana del Rey"	false	1368779	"r from alt name"	25		"n4ozbsgle13p9yywtfrz982ccj8woc9"	false	"PamD"	false	false	
+819091883	2018-01-07 10:42:27	56237371	"User talk:119.94.96.157"	3	false		false	13286072	"Warning [[Special:Contributions/119.94.96.157|119.94.96.157]] - #1"	1274		"ksohnvsbeuzwpl5vb8a3v8m18hva0a7"	false	"ClueBot NG"	false	false	
+819091914	2018-01-07 10:42:50	56237372	"Category:Ohmi Railway"	14	false		false	677153	"[[WP:AES|←]]Created page with ' [[Category:Railway companies of Japan]] [[Category:Rail transport in Shiga Prefecture]] [[Category:Seibu Group]]'"	113		"je7aw21fedbwyqsyofpisdrynsu7olr"	false	"Underbar dk"	false	false	
+819091968	2018-01-07 10:43:32	56237375	"User talk:92.226.219.222"	3	false		false	882433	"[[WP:AES|←]]Created page with '{{3rr}}~~~~'"	199		"cpm4tkzcx4hc6irr9ukbi06ogud8dtq"	false	"TastyPoutine"	false	false	
+819094036	2018-01-07 11:10:24	56237375	"User talk:92.226.219.222"	3	false		false	7611264	"[[User:AnomieBOT/docs/TemplateSubster|Substing templates]]: {{3rr}}. See [[User:AnomieBOT/docs/TemplateSubster]] for info."	1840		"artmfz8b2gxhb3pp8a5p4ksplxqfkpg"	true	"AnomieBOT"	false	false	
+819112363	2018-01-07 14:33:36	56237375	"User talk:92.226.219.222"	3	false		false	702940	"Your IP address has been blocked from editing because it has been used to [[WP:EVADE|evade a previous block]]. ([[WP:TW|TW]])"	2949		"dn9wj0n8d8pdd5lqe56uw5xamupowr1"	false	"Only"	false	false	"WP:EVADE"
+819092004	2018-01-07 10:44:01	56237376	"User:Dipayanacharya"	2	false		false	32794237	"Education"	28		"ofueugwatmmn7u73isw732neuza57gk"	false	"Dipayanacharya"	false	false	
+819092390	2018-01-07 10:49:08	56237376	"User:Dipayanacharya"	2	false		false	32794237	"School"	38		"dsz55xv96ec2uv6w9c1z7c52ipfovbw"	false	"Dipayanacharya"	false	false	
+819092066	2018-01-07 10:44:56	56237378	"BSCIC"	0	true	"Bangladesh Small and Cottage Industries Corporation"	false	21516552	"[[WP:AES|←]]Redirected page to [[Bangladesh Small and Cottage Industries Corporation]]"	65		"9ma38hak0ef1ew4fpiutxpnzd8oz1wd"	false	"Vinegarymass911"	false	false	
+819092102	2018-01-07 10:45:21	56237379	"Category:Women government ministers of Yemen"	14	false		false	754619	"[[WP:AES|←]]Created page with '{{portal|Yemen|Politics}} {{Non-diffusing subcategory|Government ministers of Yemen}}  {{Underpopulated category}}  Category:Women government ministers by nati...'"	285		"4dvakoat58bzyf5hmtthxukt29hip6n"	false	"BrownHairedGirl"	false	false	
+819092135	2018-01-07 10:45:54	56237381	"Talk:List of Morning Glories Characters"	1	false		false	410898	"[[WP:AES|←]]Created page with '{{WikiProject Fictional characters|class=List|importance=low}} {{Comicsproj|class=List|importance=low}}'"	103		"2sjrxsc7os9k9pg4su2t4rk2j8nn0h7"	false	"PRehse"	false	false	
+819092138	2018-01-07 10:45:56	56237382	"User talk:106.207.126.114"	3	false		false	13286072	"Warning [[Special:Contributions/106.207.126.114|106.207.126.114]] - #1"	1330		"3y9t5wpk6ur5jhone75rhm4wjf01fgi"	false	"ClueBot NG"	false	false	
+819092495	2018-01-07 10:50:22	56237382	"User talk:106.207.126.114"	3	false		false	31190506	"Caution: Unconstructive editing on [[List of Baahubali characters]]. ([[WP:TW|TW]])"	2355		"8wvn6vh3isyt0dorpe89lztrburgupe"	false	"HindWIKI"	false	false	

test/baseline_output/capturegroup_regextest_0.tsv

@@ -1,27 +1,27 @@
-"revid"	"date_time"	"articleid"	"title"	"namespace"	"deleted"	"editorid"	"edit_summary"	"text_chars"	"reverteds"	"sha1"	"minor"	"editor"	"anon"	"revert"	"li_cheval"	"three_letter"	"three_number"	"three_cat"
-819091731	2018-01-07 10:40:58	56237363	"User talk:86.139.142.254"	3	false	3742946	"Your IP address has been blocked from editing because it has been used to [[WP:EVADE|evade a previous block]]. ([[WP:TW|TW]])"	1141		"135nz8q6lfam6cojla7azb7k5alx3t3"	false	"NinjaRobotPirate"	false	false		"has, has"		
-819091755	2018-01-07 10:41:10	56237364	"User talk:Kavin kavitha"	3	false	32792125	"[[WP:AES|←]]Created page with ''''''Kavin (Tamil. கவின்) is a masculine given name, which is Tamil for ""beauty"", ""grace"", ""fairness"" or ""comeliness""Kavin is born on 01 /12/2001 at Sa...'"	663		"0pwezjc6yopz0smc8al6ogc4fax5bwo"	false	"Kavin kavitha"	false	false		"AES, for"	"01, 12, 2001"	
-819091788	2018-01-07 10:41:26	56237365	"User talk:Dr.vivek163"	3	false	32621254	"/* Regarding Merger discussion */ new section"	399		"sz3t2ap7z8bpkdvdvi195f3i35949bv"	false	"Amicable always"	false	false		"new"		
-819091796	2018-01-07 10:41:31	56237366	"User talk:Twistorl"	3	false	13286072	"Warning [[Special:Contributions/Twistorl|Twistorl]] - #1"	1260		"r6s5j8j3iykenrhuhpnkpsmmd71vubf"	false	"ClueBot NG"	false	false			"1"	
-819091825	2018-01-07 10:41:51	56237368	"Kom Firin"	0	false	8409334	"[[WP:AES|←]]Created page with '[[File:Stele 67.119 Brooklyn.jpg|thumb|Stele of the [[Libu#Great Chiefs of the Libu|Chief of the Libu]] Titaru, a contemporary of pharaoh [[Shoshenq V]] of the [...'TestCaseB and you're a Tor node "	2249		"tf5qz2yaswx61zrlm9ovxzuhl7r2dc4"	false	"Khruner"	false	false		"AES, jpg, the, the, the, the, and, you, Tor"	"67, 119"	
-822610647	2018-01-27 12:16:02	56237368	"Kom Firin"	0	false	8409334	"/* History */ typo"	2230		"e6oa4g0qv64icdaq26uu1zzbyr5hcbh"	true	"Khruner"	false	false				
-819091844	2018-01-07 10:42:05	56237369	"User:Editingaccount1994/sandbox"	2	false	32794215	"[[WP:AES|←]]Created page with '{{User sandbox}} <!-- EDIT BELOW THIS LINE --> {{voir homonymes|Chevalier}} {{Infobox Artiste  | nom                = Li Chevalier  | autres noms        =   | im...'"	27840		"0fyvyh2a8xu41gt8obr34oba0bfixj6"	false	"Editingaccount1994"	false	false	"Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier"	"AES, nom"		
-819093984	2018-01-07 11:09:52	56237369	"User:Editingaccount1994/sandbox"	2	false	7611264	"[[User:AnomieBOT/docs/TemplateSubster|Substing templates]]: {{Lien web}}. See [[User:AnomieBOT/docs/TemplateSubster]] for info."	27787		"8gy52aolt5rg3eaketwj5v7eiw0apv2"	true	"AnomieBOT"	false	false	"Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier"	"web, See, for"		
-820064189	2018-01-12 21:45:50	56237369	"User:Editingaccount1994/sandbox"	2	false	12406635	"Orphan per [[WP:TFD|TFD outcome]]"	27784		"he8ydemaanxlrpftqxkez8jfpge1fsj"	true	"SporkBot"	false	false	"Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier"	"per, TFD, TFD"		
-820078679	2018-01-12 23:28:11	56237369	"User:Editingaccount1994/sandbox"	2	false	12406635	"Replace template per [[Wikipedia:Templates for discussion/Log/2010 June 13|TFD outcome]]; no change in content"	27783		"0to17w9rth3url8n7gvucdtobybdq5h"	true	"SporkBot"	false	false	"Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier"	"per, for, Log, TFD"	"2010, 13"	
-820078733	2018-01-12 23:28:39	56237369	"User:Editingaccount1994/sandbox"	2	false	12406635	"Replace template per [[Wikipedia:Templates for discussion/Log/2011 February 17|TFD outcome]]; no change in content"	27782		"531dizmmloyxffbkdr5vph7owh921eg"	true	"SporkBot"	false	false	"Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier"	"per, for, Log, TFD"	"2011, 17"	
-820177382	2018-01-13 13:45:33	56237369	"User:Editingaccount1994/sandbox"	2	false	13791031	"translate TestCaseD if you are from tor you need neutral point of view "	27757		"nik9p2u2fuk4yazjxt8ymbicxv5qid9"	false	"Frietjes"	false	false	"Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier"	"you, are, tor, you"		
-822038928	2018-01-24 01:35:22	56237369	"User:Editingaccount1994/sandbox"	2	false	2304267	"Removing [[:c:File:Li_Chevalier_Art_Studio.jpg|Li_Chevalier_Art_Studio.jpg]], it has been deleted from Commons by [[:c:User:JuTa|JuTa]] because: [[:c:COM:OTRS|No permission]] since 16 January 2018."	27667		"gwk6pampl8si1v5pv3kwgteg710sfw3"	false	"CommonsDelinker"	false	false	"Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier"	"jpg, jpg, has, COM"	"16, 2018"	
-819091874	2018-01-07 10:42:20	56237370	"Anita del Rey"	0	false	1368779	"r from alt name"	25		"n4ozbsgle13p9yywtfrz982ccj8woc9"	false	"PamD"	false	false		"alt"		
-819091883	2018-01-07 10:42:27	56237371	"User talk:119.94.96.157"	3	false	13286072	"Warning [[Special:Contributions/119.94.96.157|119.94.96.157]] - #1"	1274		"ksohnvsbeuzwpl5vb8a3v8m18hva0a7"	false	"ClueBot NG"	false	false			"119, 94, 96, 157, 119, 94, 96, 157, 1"	
-819091914	2018-01-07 10:42:50	56237372	"Category:Ohmi Railway"	14	false	677153	"[[WP:AES|←]]Created page with ' [[Category:Railway companies of Japan]] [[Category:Rail transport in Shiga Prefecture]] [[Category:Seibu Group]]'"	113		"je7aw21fedbwyqsyofpisdrynsu7olr"	false	"Underbar dk"	false	false		"AES"		
-819091968	2018-01-07 10:43:32	56237375	"User talk:92.226.219.222"	3	false	882433	"[[WP:AES|←]]Created page with '{{3rr}}~~~~'"	199		"cpm4tkzcx4hc6irr9ukbi06ogud8dtq"	false	"TastyPoutine"	false	false		"AES"		
-819094036	2018-01-07 11:10:24	56237375	"User talk:92.226.219.222"	3	false	7611264	"[[User:AnomieBOT/docs/TemplateSubster|Substing templates]]: {{3rr}}. See [[User:AnomieBOT/docs/TemplateSubster]] for info."	1840		"artmfz8b2gxhb3pp8a5p4ksplxqfkpg"	true	"AnomieBOT"	false	false		"See, for"		
-819112363	2018-01-07 14:33:36	56237375	"User talk:92.226.219.222"	3	false	702940	"Your IP address has been blocked from editing because it has been used to [[WP:EVADE|evade a previous block]]. ([[WP:TW|TW]])"	2949		"dn9wj0n8d8pdd5lqe56uw5xamupowr1"	false	"Only"	false	false		"has, has"		
-819092004	2018-01-07 10:44:01	56237376	"User:Dipayanacharya"	2	false	32794237	"Education"	28		"ofueugwatmmn7u73isw732neuza57gk"	false	"Dipayanacharya"	false	false				
-819092390	2018-01-07 10:49:08	56237376	"User:Dipayanacharya"	2	false	32794237	"School"	38		"dsz55xv96ec2uv6w9c1z7c52ipfovbw"	false	"Dipayanacharya"	false	false				
-819092066	2018-01-07 10:44:56	56237378	"BSCIC"	0	false	21516552	"[[WP:AES|←]]Redirected page to [[Bangladesh Small and Cottage Industries Corporation]]"	65		"9ma38hak0ef1ew4fpiutxpnzd8oz1wd"	false	"Vinegarymass911"	false	false		"AES, and"		
-819092102	2018-01-07 10:45:21	56237379	"Category:Women government ministers of Yemen"	14	false	754619	"[[WP:AES|←]]Created page with '{{portal|Yemen|Politics}} {{Non-diffusing subcategory|Government ministers of Yemen}}  {{Underpopulated category}}  Category:Women government ministers by nati...'"	285		"4dvakoat58bzyf5hmtthxukt29hip6n"	false	"BrownHairedGirl"	false	false		"AES, Non"		
-819092135	2018-01-07 10:45:54	56237381	"Talk:List of Morning Glories Characters"	1	false	410898	"[[WP:AES|←]]Created page with '{{WikiProject Fictional characters|class=List|importance=low}} {{Comicsproj|class=List|importance=low}}'"	103		"2sjrxsc7os9k9pg4su2t4rk2j8nn0h7"	false	"PRehse"	false	false		"AES, low, low"		
-819092138	2018-01-07 10:45:56	56237382	"User talk:106.207.126.114"	3	false	13286072	"Warning [[Special:Contributions/106.207.126.114|106.207.126.114]] - #1"	1330		"3y9t5wpk6ur5jhone75rhm4wjf01fgi"	false	"ClueBot NG"	false	false			"106, 207, 126, 114, 106, 207, 126, 114, 1"	
-819092495	2018-01-07 10:50:22	56237382	"User talk:106.207.126.114"	3	false	31190506	"Caution: Unconstructive editing on [[List of Baahubali characters]]. ([[WP:TW|TW]])"	2355		"8wvn6vh3isyt0dorpe89lztrburgupe"	false	"HindWIKI"	false	false				
+"revid"	"date_time"	"articleid"	"title"	"namespace"	"revision_is_redirect"	"revision_redirect_target"	"deleted"	"editorid"	"edit_summary"	"text_chars"	"reverteds"	"sha1"	"minor"	"editor"	"anon"	"revert"	"li_cheval"	"three_letter"	"three_number"	"three_cat"
+819091731	2018-01-07 10:40:58	56237363	"User talk:86.139.142.254"	3	false		false	3742946	"Your IP address has been blocked from editing because it has been used to [[WP:EVADE|evade a previous block]]. ([[WP:TW|TW]])"	1141		"135nz8q6lfam6cojla7azb7k5alx3t3"	false	"NinjaRobotPirate"	false	false		"has, has"		
+819091755	2018-01-07 10:41:10	56237364	"User talk:Kavin kavitha"	3	false		false	32792125	"[[WP:AES|←]]Created page with ''''''Kavin (Tamil. கவின்) is a masculine given name, which is Tamil for ""beauty"", ""grace"", ""fairness"" or ""comeliness""Kavin is born on 01 /12/2001 at Sa...'"	663		"0pwezjc6yopz0smc8al6ogc4fax5bwo"	false	"Kavin kavitha"	false	false		"AES, for"	"01, 12, 2001"	
+819091788	2018-01-07 10:41:26	56237365	"User talk:Dr.vivek163"	3	false		false	32621254	"/* Regarding Merger discussion */ new section"	399		"sz3t2ap7z8bpkdvdvi195f3i35949bv"	false	"Amicable always"	false	false		"new"		
+819091796	2018-01-07 10:41:31	56237366	"User talk:Twistorl"	3	false		false	13286072	"Warning [[Special:Contributions/Twistorl|Twistorl]] - #1"	1260		"r6s5j8j3iykenrhuhpnkpsmmd71vubf"	false	"ClueBot NG"	false	false			"1"	
+819091825	2018-01-07 10:41:51	56237368	"Kom Firin"	0	false		false	8409334	"[[WP:AES|←]]Created page with '[[File:Stele 67.119 Brooklyn.jpg|thumb|Stele of the [[Libu#Great Chiefs of the Libu|Chief of the Libu]] Titaru, a contemporary of pharaoh [[Shoshenq V]] of the [...'TestCaseB and you're a Tor node "	2249		"tf5qz2yaswx61zrlm9ovxzuhl7r2dc4"	false	"Khruner"	false	false		"AES, jpg, the, the, the, the, and, you, Tor"	"67, 119"	
+822610647	2018-01-27 12:16:02	56237368	"Kom Firin"	0	false		false	8409334	"/* History */ typo"	2230		"e6oa4g0qv64icdaq26uu1zzbyr5hcbh"	true	"Khruner"	false	false				
+819091844	2018-01-07 10:42:05	56237369	"User:Editingaccount1994/sandbox"	2	false		false	32794215	"[[WP:AES|←]]Created page with '{{User sandbox}} <!-- EDIT BELOW THIS LINE --> {{voir homonymes|Chevalier}} {{Infobox Artiste  | nom                = Li Chevalier  | autres noms        =   | im...'"	27840		"0fyvyh2a8xu41gt8obr34oba0bfixj6"	false	"Editingaccount1994"	false	false	"Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier"	"AES, nom"		
+819093984	2018-01-07 11:09:52	56237369	"User:Editingaccount1994/sandbox"	2	false		false	7611264	"[[User:AnomieBOT/docs/TemplateSubster|Substing templates]]: {{Lien web}}. See [[User:AnomieBOT/docs/TemplateSubster]] for info."	27787		"8gy52aolt5rg3eaketwj5v7eiw0apv2"	true	"AnomieBOT"	false	false	"Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier"	"web, See, for"		
+820064189	2018-01-12 21:45:50	56237369	"User:Editingaccount1994/sandbox"	2	false		false	12406635	"Orphan per [[WP:TFD|TFD outcome]]"	27784		"he8ydemaanxlrpftqxkez8jfpge1fsj"	true	"SporkBot"	false	false	"Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier"	"per, TFD, TFD"		
+820078679	2018-01-12 23:28:11	56237369	"User:Editingaccount1994/sandbox"	2	false		false	12406635	"Replace template per [[Wikipedia:Templates for discussion/Log/2010 June 13|TFD outcome]]; no change in content"	27783		"0to17w9rth3url8n7gvucdtobybdq5h"	true	"SporkBot"	false	false	"Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier"	"per, for, Log, TFD"	"2010, 13"	
+820078733	2018-01-12 23:28:39	56237369	"User:Editingaccount1994/sandbox"	2	false		false	12406635	"Replace template per [[Wikipedia:Templates for discussion/Log/2011 February 17|TFD outcome]]; no change in content"	27782		"531dizmmloyxffbkdr5vph7owh921eg"	true	"SporkBot"	false	false	"Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier"	"per, for, Log, TFD"	"2011, 17"	
+820177382	2018-01-13 13:45:33	56237369	"User:Editingaccount1994/sandbox"	2	false		false	13791031	"translate TestCaseD if you are from tor you need neutral point of view "	27757		"nik9p2u2fuk4yazjxt8ymbicxv5qid9"	false	"Frietjes"	false	false	"Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier"	"you, are, tor, you"		
+822038928	2018-01-24 01:35:22	56237369	"User:Editingaccount1994/sandbox"	2	false		false	2304267	"Removing [[:c:File:Li_Chevalier_Art_Studio.jpg|Li_Chevalier_Art_Studio.jpg]], it has been deleted from Commons by [[:c:User:JuTa|JuTa]] because: [[:c:COM:OTRS|No permission]] since 16 January 2018."	27667		"gwk6pampl8si1v5pv3kwgteg710sfw3"	false	"CommonsDelinker"	false	false	"Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier, Li Chevalier"	"jpg, jpg, has, COM"	"16, 2018"	
+819091874	2018-01-07 10:42:20	56237370	"Anita del Rey"	0	true	"Ana del Rey"	false	1368779	"r from alt name"	25		"n4ozbsgle13p9yywtfrz982ccj8woc9"	false	"PamD"	false	false		"alt"		
+819091883	2018-01-07 10:42:27	56237371	"User talk:119.94.96.157"	3	false		false	13286072	"Warning [[Special:Contributions/119.94.96.157|119.94.96.157]] - #1"	1274		"ksohnvsbeuzwpl5vb8a3v8m18hva0a7"	false	"ClueBot NG"	false	false			"119, 94, 96, 157, 119, 94, 96, 157, 1"	
+819091914	2018-01-07 10:42:50	56237372	"Category:Ohmi Railway"	14	false		false	677153	"[[WP:AES|←]]Created page with ' [[Category:Railway companies of Japan]] [[Category:Rail transport in Shiga Prefecture]] [[Category:Seibu Group]]'"	113		"je7aw21fedbwyqsyofpisdrynsu7olr"	false	"Underbar dk"	false	false		"AES"		
+819091968	2018-01-07 10:43:32	56237375	"User talk:92.226.219.222"	3	false		false	882433	"[[WP:AES|←]]Created page with '{{3rr}}~~~~'"	199		"cpm4tkzcx4hc6irr9ukbi06ogud8dtq"	false	"TastyPoutine"	false	false		"AES"		
+819094036	2018-01-07 11:10:24	56237375	"User talk:92.226.219.222"	3	false		false	7611264	"[[User:AnomieBOT/docs/TemplateSubster|Substing templates]]: {{3rr}}. See [[User:AnomieBOT/docs/TemplateSubster]] for info."	1840		"artmfz8b2gxhb3pp8a5p4ksplxqfkpg"	true	"AnomieBOT"	false	false		"See, for"		
+819112363	2018-01-07 14:33:36	56237375	"User talk:92.226.219.222"	3	false		false	702940	"Your IP address has been blocked from editing because it has been used to [[WP:EVADE|evade a previous block]]. ([[WP:TW|TW]])"	2949		"dn9wj0n8d8pdd5lqe56uw5xamupowr1"	false	"Only"	false	false		"has, has"		
+819092004	2018-01-07 10:44:01	56237376	"User:Dipayanacharya"	2	false		false	32794237	"Education"	28		"ofueugwatmmn7u73isw732neuza57gk"	false	"Dipayanacharya"	false	false				
+819092390	2018-01-07 10:49:08	56237376	"User:Dipayanacharya"	2	false		false	32794237	"School"	38		"dsz55xv96ec2uv6w9c1z7c52ipfovbw"	false	"Dipayanacharya"	false	false				
+819092066	2018-01-07 10:44:56	56237378	"BSCIC"	0	true	"Bangladesh Small and Cottage Industries Corporation"	false	21516552	"[[WP:AES|←]]Redirected page to [[Bangladesh Small and Cottage Industries Corporation]]"	65		"9ma38hak0ef1ew4fpiutxpnzd8oz1wd"	false	"Vinegarymass911"	false	false		"AES, and"		
+819092102	2018-01-07 10:45:21	56237379	"Category:Women government ministers of Yemen"	14	false		false	754619	"[[WP:AES|←]]Created page with '{{portal|Yemen|Politics}} {{Non-diffusing subcategory|Government ministers of Yemen}}  {{Underpopulated category}}  Category:Women government ministers by nati...'"	285		"4dvakoat58bzyf5hmtthxukt29hip6n"	false	"BrownHairedGirl"	false	false		"AES, Non"		
+819092135	2018-01-07 10:45:54	56237381	"Talk:List of Morning Glories Characters"	1	false		false	410898	"[[WP:AES|←]]Created page with '{{WikiProject Fictional characters|class=List|importance=low}} {{Comicsproj|class=List|importance=low}}'"	103		"2sjrxsc7os9k9pg4su2t4rk2j8nn0h7"	false	"PRehse"	false	false		"AES, low, low"		
+819092138	2018-01-07 10:45:56	56237382	"User talk:106.207.126.114"	3	false		false	13286072	"Warning [[Special:Contributions/106.207.126.114|106.207.126.114]] - #1"	1330		"3y9t5wpk6ur5jhone75rhm4wjf01fgi"	false	"ClueBot NG"	false	false			"106, 207, 126, 114, 106, 207, 126, 114, 1"	
+819092495	2018-01-07 10:50:22	56237382	"User talk:106.207.126.114"	3	false		false	31190506	"Caution: Unconstructive editing on [[List of Baahubali characters]]. ([[WP:TW|TW]])"	2355		"8wvn6vh3isyt0dorpe89lztrburgupe"	false	"HindWIKI"	false	false				

test/baseline_output/capturegroup_regextest_1.tsv

@@ -1,27 +1,27 @@
-"revid"	"date_time"	"articleid"	"title"	"namespace"	"deleted"	"editorid"	"edit_summary"	"text_chars"	"reverteds"	"sha1"	"minor"	"editor"	"anon"	"revert"	"npov_npov"	"npov_neutral"	"testcase_a"	"testcase_b"	"testcase_c"	"testcase_d"
-819091731	2018-01-07 10:40:58	56237363	"User talk:86.139.142.254"	3	false	3742946	"Your IP address has been blocked from editing because it has been used to [[WP:EVADE|evade a previous block]]. ([[WP:TW|TW]])"	1141		"135nz8q6lfam6cojla7azb7k5alx3t3"	false	"NinjaRobotPirate"	false	false						
-819091755	2018-01-07 10:41:10	56237364	"User talk:Kavin kavitha"	3	false	32792125	"[[WP:AES|←]]Created page with ''''''Kavin (Tamil. கவின்) is a masculine given name, which is Tamil for ""beauty"", ""grace"", ""fairness"" or ""comeliness""Kavin is born on 01 /12/2001 at Sa...'"	663		"0pwezjc6yopz0smc8al6ogc4fax5bwo"	false	"Kavin kavitha"	false	false						
-819091788	2018-01-07 10:41:26	56237365	"User talk:Dr.vivek163"	3	false	32621254	"/* Regarding Merger discussion */ new section"	399		"sz3t2ap7z8bpkdvdvi195f3i35949bv"	false	"Amicable always"	false	false	"NPOV, NPOV"					
-819091796	2018-01-07 10:41:31	56237366	"User talk:Twistorl"	3	false	13286072	"Warning [[Special:Contributions/Twistorl|Twistorl]] - #1"	1260		"r6s5j8j3iykenrhuhpnkpsmmd71vubf"	false	"ClueBot NG"	false	false						
-819091825	2018-01-07 10:41:51	56237368	"Kom Firin"	0	false	8409334	"[[WP:AES|←]]Created page with '[[File:Stele 67.119 Brooklyn.jpg|thumb|Stele of the [[Libu#Great Chiefs of the Libu|Chief of the Libu]] Titaru, a contemporary of pharaoh [[Shoshenq V]] of the [...'TestCaseB and you're a Tor node "	2249		"tf5qz2yaswx61zrlm9ovxzuhl7r2dc4"	false	"Khruner"	false	false	"NPOV"			"TestCaseB"		
-822610647	2018-01-27 12:16:02	56237368	"Kom Firin"	0	false	8409334	"/* History */ typo"	2230		"e6oa4g0qv64icdaq26uu1zzbyr5hcbh"	true	"Khruner"	false	false						
-819091844	2018-01-07 10:42:05	56237369	"User:Editingaccount1994/sandbox"	2	false	32794215	"[[WP:AES|←]]Created page with '{{User sandbox}} <!-- EDIT BELOW THIS LINE --> {{voir homonymes|Chevalier}} {{Infobox Artiste  | nom                = Li Chevalier  | autres noms        =   | im...'"	27840		"0fyvyh2a8xu41gt8obr34oba0bfixj6"	false	"Editingaccount1994"	false	false						
-819093984	2018-01-07 11:09:52	56237369	"User:Editingaccount1994/sandbox"	2	false	7611264	"[[User:AnomieBOT/docs/TemplateSubster|Substing templates]]: {{Lien web}}. See [[User:AnomieBOT/docs/TemplateSubster]] for info."	27787		"8gy52aolt5rg3eaketwj5v7eiw0apv2"	true	"AnomieBOT"	false	false						
-820064189	2018-01-12 21:45:50	56237369	"User:Editingaccount1994/sandbox"	2	false	12406635	"Orphan per [[WP:TFD|TFD outcome]]"	27784		"he8ydemaanxlrpftqxkez8jfpge1fsj"	true	"SporkBot"	false	false						
-820078679	2018-01-12 23:28:11	56237369	"User:Editingaccount1994/sandbox"	2	false	12406635	"Replace template per [[Wikipedia:Templates for discussion/Log/2010 June 13|TFD outcome]]; no change in content"	27783		"0to17w9rth3url8n7gvucdtobybdq5h"	true	"SporkBot"	false	false						
-820078733	2018-01-12 23:28:39	56237369	"User:Editingaccount1994/sandbox"	2	false	12406635	"Replace template per [[Wikipedia:Templates for discussion/Log/2011 February 17|TFD outcome]]; no change in content"	27782		"531dizmmloyxffbkdr5vph7owh921eg"	true	"SporkBot"	false	false						
-820177382	2018-01-13 13:45:33	56237369	"User:Editingaccount1994/sandbox"	2	false	13791031	"translate TestCaseD if you are from tor you need neutral point of view "	27757		"nik9p2u2fuk4yazjxt8ymbicxv5qid9"	false	"Frietjes"	false	false						"TestCaseD"
-822038928	2018-01-24 01:35:22	56237369	"User:Editingaccount1994/sandbox"	2	false	2304267	"Removing [[:c:File:Li_Chevalier_Art_Studio.jpg|Li_Chevalier_Art_Studio.jpg]], it has been deleted from Commons by [[:c:User:JuTa|JuTa]] because: [[:c:COM:OTRS|No permission]] since 16 January 2018."	27667		"gwk6pampl8si1v5pv3kwgteg710sfw3"	false	"CommonsDelinker"	false	false						
-819091874	2018-01-07 10:42:20	56237370	"Anita del Rey"	0	false	1368779	"r from alt name"	25		"n4ozbsgle13p9yywtfrz982ccj8woc9"	false	"PamD"	false	false						
-819091883	2018-01-07 10:42:27	56237371	"User talk:119.94.96.157"	3	false	13286072	"Warning [[Special:Contributions/119.94.96.157|119.94.96.157]] - #1"	1274		"ksohnvsbeuzwpl5vb8a3v8m18hva0a7"	false	"ClueBot NG"	false	false						
-819091914	2018-01-07 10:42:50	56237372	"Category:Ohmi Railway"	14	false	677153	"[[WP:AES|←]]Created page with ' [[Category:Railway companies of Japan]] [[Category:Rail transport in Shiga Prefecture]] [[Category:Seibu Group]]'"	113		"je7aw21fedbwyqsyofpisdrynsu7olr"	false	"Underbar dk"	false	false						
-819091968	2018-01-07 10:43:32	56237375	"User talk:92.226.219.222"	3	false	882433	"[[WP:AES|←]]Created page with '{{3rr}}~~~~'"	199		"cpm4tkzcx4hc6irr9ukbi06ogud8dtq"	false	"TastyPoutine"	false	false						
-819094036	2018-01-07 11:10:24	56237375	"User talk:92.226.219.222"	3	false	7611264	"[[User:AnomieBOT/docs/TemplateSubster|Substing templates]]: {{3rr}}. See [[User:AnomieBOT/docs/TemplateSubster]] for info."	1840		"artmfz8b2gxhb3pp8a5p4ksplxqfkpg"	true	"AnomieBOT"	false	false						
-819112363	2018-01-07 14:33:36	56237375	"User talk:92.226.219.222"	3	false	702940	"Your IP address has been blocked from editing because it has been used to [[WP:EVADE|evade a previous block]]. ([[WP:TW|TW]])"	2949		"dn9wj0n8d8pdd5lqe56uw5xamupowr1"	false	"Only"	false	false						
-819092004	2018-01-07 10:44:01	56237376	"User:Dipayanacharya"	2	false	32794237	"Education"	28		"ofueugwatmmn7u73isw732neuza57gk"	false	"Dipayanacharya"	false	false						
-819092390	2018-01-07 10:49:08	56237376	"User:Dipayanacharya"	2	false	32794237	"School"	38		"dsz55xv96ec2uv6w9c1z7c52ipfovbw"	false	"Dipayanacharya"	false	false						
-819092066	2018-01-07 10:44:56	56237378	"BSCIC"	0	false	21516552	"[[WP:AES|←]]Redirected page to [[Bangladesh Small and Cottage Industries Corporation]]"	65		"9ma38hak0ef1ew4fpiutxpnzd8oz1wd"	false	"Vinegarymass911"	false	false						
-819092102	2018-01-07 10:45:21	56237379	"Category:Women government ministers of Yemen"	14	false	754619	"[[WP:AES|←]]Created page with '{{portal|Yemen|Politics}} {{Non-diffusing subcategory|Government ministers of Yemen}}  {{Underpopulated category}}  Category:Women government ministers by nati...'"	285		"4dvakoat58bzyf5hmtthxukt29hip6n"	false	"BrownHairedGirl"	false	false						
-819092135	2018-01-07 10:45:54	56237381	"Talk:List of Morning Glories Characters"	1	false	410898	"[[WP:AES|←]]Created page with '{{WikiProject Fictional characters|class=List|importance=low}} {{Comicsproj|class=List|importance=low}}'"	103		"2sjrxsc7os9k9pg4su2t4rk2j8nn0h7"	false	"PRehse"	false	false						
-819092138	2018-01-07 10:45:56	56237382	"User talk:106.207.126.114"	3	false	13286072	"Warning [[Special:Contributions/106.207.126.114|106.207.126.114]] - #1"	1330		"3y9t5wpk6ur5jhone75rhm4wjf01fgi"	false	"ClueBot NG"	false	false						
-819092495	2018-01-07 10:50:22	56237382	"User talk:106.207.126.114"	3	false	31190506	"Caution: Unconstructive editing on [[List of Baahubali characters]]. ([[WP:TW|TW]])"	2355		"8wvn6vh3isyt0dorpe89lztrburgupe"	false	"HindWIKI"	false	false						
+"revid"	"date_time"	"articleid"	"title"	"namespace"	"revision_is_redirect"	"revision_redirect_target"	"deleted"	"editorid"	"edit_summary"	"text_chars"	"reverteds"	"sha1"	"minor"	"editor"	"anon"	"revert"	"npov_npov"	"npov_neutral"	"testcase_a"	"testcase_b"	"testcase_c"	"testcase_d"
+819091731	2018-01-07 10:40:58	56237363	"User talk:86.139.142.254"	3	false		false	3742946	"Your IP address has been blocked from editing because it has been used to [[WP:EVADE|evade a previous block]]. ([[WP:TW|TW]])"	1141		"135nz8q6lfam6cojla7azb7k5alx3t3"	false	"NinjaRobotPirate"	false	false						
+819091755	2018-01-07 10:41:10	56237364	"User talk:Kavin kavitha"	3	false		false	32792125	"[[WP:AES|←]]Created page with ''''''Kavin (Tamil. கவின்) is a masculine given name, which is Tamil for ""beauty"", ""grace"", ""fairness"" or ""comeliness""Kavin is born on 01 /12/2001 at Sa...'"	663		"0pwezjc6yopz0smc8al6ogc4fax5bwo"	false	"Kavin kavitha"	false	false						
+819091788	2018-01-07 10:41:26	56237365	"User talk:Dr.vivek163"	3	false		false	32621254	"/* Regarding Merger discussion */ new section"	399		"sz3t2ap7z8bpkdvdvi195f3i35949bv"	false	"Amicable always"	false	false	"NPOV, NPOV"					
+819091796	2018-01-07 10:41:31	56237366	"User talk:Twistorl"	3	false		false	13286072	"Warning [[Special:Contributions/Twistorl|Twistorl]] - #1"	1260		"r6s5j8j3iykenrhuhpnkpsmmd71vubf"	false	"ClueBot NG"	false	false						
+819091825	2018-01-07 10:41:51	56237368	"Kom Firin"	0	false		false	8409334	"[[WP:AES|←]]Created page with '[[File:Stele 67.119 Brooklyn.jpg|thumb|Stele of the [[Libu#Great Chiefs of the Libu|Chief of the Libu]] Titaru, a contemporary of pharaoh [[Shoshenq V]] of the [...'TestCaseB and you're a Tor node "	2249		"tf5qz2yaswx61zrlm9ovxzuhl7r2dc4"	false	"Khruner"	false	false	"NPOV"			"TestCaseB"		
+822610647	2018-01-27 12:16:02	56237368	"Kom Firin"	0	false		false	8409334	"/* History */ typo"	2230		"e6oa4g0qv64icdaq26uu1zzbyr5hcbh"	true	"Khruner"	false	false						
+819091844	2018-01-07 10:42:05	56237369	"User:Editingaccount1994/sandbox"	2	false		false	32794215	"[[WP:AES|←]]Created page with '{{User sandbox}} <!-- EDIT BELOW THIS LINE --> {{voir homonymes|Chevalier}} {{Infobox Artiste  | nom                = Li Chevalier  | autres noms        =   | im...'"	27840		"0fyvyh2a8xu41gt8obr34oba0bfixj6"	false	"Editingaccount1994"	false	false						
+819093984	2018-01-07 11:09:52	56237369	"User:Editingaccount1994/sandbox"	2	false		false	7611264	"[[User:AnomieBOT/docs/TemplateSubster|Substing templates]]: {{Lien web}}. See [[User:AnomieBOT/docs/TemplateSubster]] for info."	27787		"8gy52aolt5rg3eaketwj5v7eiw0apv2"	true	"AnomieBOT"	false	false						
+820064189	2018-01-12 21:45:50	56237369	"User:Editingaccount1994/sandbox"	2	false		false	12406635	"Orphan per [[WP:TFD|TFD outcome]]"	27784		"he8ydemaanxlrpftqxkez8jfpge1fsj"	true	"SporkBot"	false	false						
+820078679	2018-01-12 23:28:11	56237369	"User:Editingaccount1994/sandbox"	2	false		false	12406635	"Replace template per [[Wikipedia:Templates for discussion/Log/2010 June 13|TFD outcome]]; no change in content"	27783		"0to17w9rth3url8n7gvucdtobybdq5h"	true	"SporkBot"	false	false						
+820078733	2018-01-12 23:28:39	56237369	"User:Editingaccount1994/sandbox"	2	false		false	12406635	"Replace template per [[Wikipedia:Templates for discussion/Log/2011 February 17|TFD outcome]]; no change in content"	27782		"531dizmmloyxffbkdr5vph7owh921eg"	true	"SporkBot"	false	false						
+820177382	2018-01-13 13:45:33	56237369	"User:Editingaccount1994/sandbox"	2	false		false	13791031	"translate TestCaseD if you are from tor you need neutral point of view "	27757		"nik9p2u2fuk4yazjxt8ymbicxv5qid9"	false	"Frietjes"	false	false						"TestCaseD"
+822038928	2018-01-24 01:35:22	56237369	"User:Editingaccount1994/sandbox"	2	false		false	2304267	"Removing [[:c:File:Li_Chevalier_Art_Studio.jpg|Li_Chevalier_Art_Studio.jpg]], it has been deleted from Commons by [[:c:User:JuTa|JuTa]] because: [[:c:COM:OTRS|No permission]] since 16 January 2018."	27667		"gwk6pampl8si1v5pv3kwgteg710sfw3"	false	"CommonsDelinker"	false	false						
+819091874	2018-01-07 10:42:20	56237370	"Anita del Rey"	0	true	"Ana del Rey"	false	1368779	"r from alt name"	25		"n4ozbsgle13p9yywtfrz982ccj8woc9"	false	"PamD"	false	false						
+819091883	2018-01-07 10:42:27	56237371	"User talk:119.94.96.157"	3	false		false	13286072	"Warning [[Special:Contributions/119.94.96.157|119.94.96.157]] - #1"	1274		"ksohnvsbeuzwpl5vb8a3v8m18hva0a7"	false	"ClueBot NG"	false	false						
+819091914	2018-01-07 10:42:50	56237372	"Category:Ohmi Railway"	14	false		false	677153	"[[WP:AES|←]]Created page with ' [[Category:Railway companies of Japan]] [[Category:Rail transport in Shiga Prefecture]] [[Category:Seibu Group]]'"	113		"je7aw21fedbwyqsyofpisdrynsu7olr"	false	"Underbar dk"	false	false						
+819091968	2018-01-07 10:43:32	56237375	"User talk:92.226.219.222"	3	false		false	882433	"[[WP:AES|←]]Created page with '{{3rr}}~~~~'"	199		"cpm4tkzcx4hc6irr9ukbi06ogud8dtq"	false	"TastyPoutine"	false	false						
+819094036	2018-01-07 11:10:24	56237375	"User talk:92.226.219.222"	3	false		false	7611264	"[[User:AnomieBOT/docs/TemplateSubster|Substing templates]]: {{3rr}}. See [[User:AnomieBOT/docs/TemplateSubster]] for info."	1840		"artmfz8b2gxhb3pp8a5p4ksplxqfkpg"	true	"AnomieBOT"	false	false						
+819112363	2018-01-07 14:33:36	56237375	"User talk:92.226.219.222"	3	false		false	702940	"Your IP address has been blocked from editing because it has been used to [[WP:EVADE|evade a previous block]]. ([[WP:TW|TW]])"	2949		"dn9wj0n8d8pdd5lqe56uw5xamupowr1"	false	"Only"	false	false						
+819092004	2018-01-07 10:44:01	56237376	"User:Dipayanacharya"	2	false		false	32794237	"Education"	28		"ofueugwatmmn7u73isw732neuza57gk"	false	"Dipayanacharya"	false	false						
+819092390	2018-01-07 10:49:08	56237376	"User:Dipayanacharya"	2	false		false	32794237	"School"	38		"dsz55xv96ec2uv6w9c1z7c52ipfovbw"	false	"Dipayanacharya"	false	false						
+819092066	2018-01-07 10:44:56	56237378	"BSCIC"	0	true	"Bangladesh Small and Cottage Industries Corporation"	false	21516552	"[[WP:AES|←]]Redirected page to [[Bangladesh Small and Cottage Industries Corporation]]"	65		"9ma38hak0ef1ew4fpiutxpnzd8oz1wd"	false	"Vinegarymass911"	false	false						
+819092102	2018-01-07 10:45:21	56237379	"Category:Women government ministers of Yemen"	14	false		false	754619	"[[WP:AES|←]]Created page with '{{portal|Yemen|Politics}} {{Non-diffusing subcategory|Government ministers of Yemen}}  {{Underpopulated category}}  Category:Women government ministers by nati...'"	285		"4dvakoat58bzyf5hmtthxukt29hip6n"	false	"BrownHairedGirl"	false	false						
+819092135	2018-01-07 10:45:54	56237381	"Talk:List of Morning Glories Characters"	1	false		false	410898	"[[WP:AES|←]]Created page with '{{WikiProject Fictional characters|class=List|importance=low}} {{Comicsproj|class=List|importance=low}}'"	103		"2sjrxsc7os9k9pg4su2t4rk2j8nn0h7"	false	"PRehse"	false	false						
+819092138	2018-01-07 10:45:56	56237382	"User talk:106.207.126.114"	3	false		false	13286072	"Warning [[Special:Contributions/106.207.126.114|106.207.126.114]] - #1"	1330		"3y9t5wpk6ur5jhone75rhm4wjf01fgi"	false	"ClueBot NG"	false	false						
+819092495	2018-01-07 10:50:22	56237382	"User talk:106.207.126.114"	3	false		false	31190506	"Caution: Unconstructive editing on [[List of Baahubali characters]]. ([[WP:TW|TW]])"	2355		"8wvn6vh3isyt0dorpe89lztrburgupe"	false	"HindWIKI"	false	false						

test/baseline_output/regextest.tsv

@@ -1,27 +0,0 @@
-anon	articleid	chev_com	date_time	deleted	editor	editorid	minor	namespace	revert	reverteds	revid	sha1	text_chars	title	warning	wiki_welcome
-FALSE	56237363		2018-01-07 10:40:58	FALSE	"NinjaRobotPirate"	3742946	FALSE	3	FALSE		819091731	135nz8q6lfam6cojla7azb7k5alx3t3	1141	"User talk:86.139.142.254"		
-FALSE	56237364		2018-01-07 10:41:10	FALSE	"Kavin kavitha"	32792125	FALSE	3	FALSE		819091755	0pwezjc6yopz0smc8al6ogc4fax5bwo	663	"User talk:Kavin kavitha"		
-FALSE	56237365		2018-01-07 10:41:26	FALSE	"Amicable always"	32621254	FALSE	3	FALSE		819091788	sz3t2ap7z8bpkdvdvi195f3i35949bv	399	"User talk:Dr.vivek163"		
-FALSE	56237366		2018-01-07 10:41:31	FALSE	"ClueBot NG"	13286072	FALSE	3	FALSE		819091796	r6s5j8j3iykenrhuhpnkpsmmd71vubf	1260	"User talk:Twistorl"	Warning	welcome to Wikipedia
-FALSE	56237368		2018-01-07 10:41:51	FALSE	"Khruner"	8409334	FALSE	0	FALSE		819091825	tf5qz2yaswx61zrlm9ovxzuhl7r2dc4	2249	"Kom Firin"		
-FALSE	56237368		2018-01-27 12:16:02	FALSE	"Khruner"	8409334	TRUE	0	FALSE		822610647	e6oa4g0qv64icdaq26uu1zzbyr5hcbh	2230	"Kom Firin"		
-FALSE	56237369	Chevalier, Chevalier	2018-01-07 10:42:05	FALSE	"Editingaccount1994"	32794215	FALSE	2	FALSE		819091844	0fyvyh2a8xu41gt8obr34oba0bfixj6	27840	"User:Editingaccount1994/sandbox"		
-FALSE	56237369		2018-01-07 11:09:52	FALSE	"AnomieBOT"	7611264	TRUE	2	FALSE		819093984	8gy52aolt5rg3eaketwj5v7eiw0apv2	27787	"User:Editingaccount1994/sandbox"		
-FALSE	56237369		2018-01-12 21:45:50	FALSE	"SporkBot"	12406635	TRUE	2	FALSE		820064189	he8ydemaanxlrpftqxkez8jfpge1fsj	27784	"User:Editingaccount1994/sandbox"		
-FALSE	56237369		2018-01-12 23:28:11	FALSE	"SporkBot"	12406635	TRUE	2	FALSE		820078679	0to17w9rth3url8n7gvucdtobybdq5h	27783	"User:Editingaccount1994/sandbox"		
-FALSE	56237369		2018-01-12 23:28:39	FALSE	"SporkBot"	12406635	TRUE	2	FALSE		820078733	531dizmmloyxffbkdr5vph7owh921eg	27782	"User:Editingaccount1994/sandbox"		
-FALSE	56237369		2018-01-13 13:45:33	FALSE	"Frietjes"	13791031	FALSE	2	FALSE		820177382	nik9p2u2fuk4yazjxt8ymbicxv5qid9	27757	"User:Editingaccount1994/sandbox"		
-FALSE	56237369	Chevalier, Chevalier	2018-01-24 01:35:22	FALSE	"CommonsDelinker"	2304267	FALSE	2	FALSE		822038928	gwk6pampl8si1v5pv3kwgteg710sfw3	27667	"User:Editingaccount1994/sandbox"		
-FALSE	56237370		2018-01-07 10:42:20	FALSE	"PamD"	1368779	FALSE	0	FALSE		819091874	n4ozbsgle13p9yywtfrz982ccj8woc9	25	"Anita del Rey"		
-FALSE	56237371		2018-01-07 10:42:27	FALSE	"ClueBot NG"	13286072	FALSE	3	FALSE		819091883	ksohnvsbeuzwpl5vb8a3v8m18hva0a7	1274	"User talk:119.94.96.157"	Warning	welcome to Wikipedia
-FALSE	56237372		2018-01-07 10:42:50	FALSE	"Underbar dk"	677153	FALSE	14	FALSE		819091914	je7aw21fedbwyqsyofpisdrynsu7olr	113	"Category:Ohmi Railway"		
-FALSE	56237375		2018-01-07 10:43:32	FALSE	"TastyPoutine"	882433	FALSE	3	FALSE		819091968	cpm4tkzcx4hc6irr9ukbi06ogud8dtq	199	"User talk:92.226.219.222"		
-FALSE	56237375		2018-01-07 11:10:24	FALSE	"AnomieBOT"	7611264	TRUE	3	FALSE		819094036	artmfz8b2gxhb3pp8a5p4ksplxqfkpg	1840	"User talk:92.226.219.222"		
-FALSE	56237375		2018-01-07 14:33:36	FALSE	"Only"	702940	FALSE	3	FALSE		819112363	dn9wj0n8d8pdd5lqe56uw5xamupowr1	2949	"User talk:92.226.219.222"		
-FALSE	56237376		2018-01-07 10:44:01	FALSE	"Dipayanacharya"	32794237	FALSE	2	FALSE		819092004	ofueugwatmmn7u73isw732neuza57gk	28	"User:Dipayanacharya"		
-FALSE	56237376		2018-01-07 10:49:08	FALSE	"Dipayanacharya"	32794237	FALSE	2	FALSE		819092390	dsz55xv96ec2uv6w9c1z7c52ipfovbw	38	"User:Dipayanacharya"		
-FALSE	56237378		2018-01-07 10:44:56	FALSE	"Vinegarymass911"	21516552	FALSE	0	FALSE		819092066	9ma38hak0ef1ew4fpiutxpnzd8oz1wd	65	"BSCIC"		
-FALSE	56237379		2018-01-07 10:45:21	FALSE	"BrownHairedGirl"	754619	FALSE	14	FALSE		819092102	4dvakoat58bzyf5hmtthxukt29hip6n	285	"Category:Women government ministers of Yemen"		
-FALSE	56237381		2018-01-07 10:45:54	FALSE	"PRehse"	410898	FALSE	1	FALSE		819092135	2sjrxsc7os9k9pg4su2t4rk2j8nn0h7	103	"Talk:List of Morning Glories Characters"		
-FALSE	56237382		2018-01-07 10:45:56	FALSE	"ClueBot NG"	13286072	FALSE	3	FALSE		819092138	3y9t5wpk6ur5jhone75rhm4wjf01fgi	1330	"User talk:106.207.126.114"	Warning	welcome to Wikipedia
-FALSE	56237382		2018-01-07 10:50:22	FALSE	"HindWIKI"	31190506	FALSE	3	FALSE		819092495	8wvn6vh3isyt0dorpe89lztrburgupe	2355	"User talk:106.207.126.114"		welcome to Wikipedia

test/test_resume.py

@@ -9,16 +9,13 @@ import time
 
 import pytest
 
-from wikiq.resume import (
-    get_checkpoint_path,
-    read_checkpoint,
-)
 from wikiq_test_utils import (
     SAILORMOON,
     TEST_DIR,
     TEST_OUTPUT_DIR,
     WIKIQ,
     WikiqTester,
+    requires_pywikidiff2,
 )
 
 
@@ -33,7 +30,7 @@ def read_jsonl(filepath):
 
 
 def test_resume():
-    """Test that --resume properly resumes processing from the last checkpoint."""
+    """Test that --resume properly resumes processing from the last line of output."""
     import pandas as pd
     from pandas.testing import assert_frame_equal
 
@@ -57,10 +54,6 @@ def test_resume():
         for row in full_rows[:middle_idx + 1]:
             f.write(json.dumps(row) + "\n")
 
-    checkpoint_path = get_checkpoint_path(partial_output_path)
-    with open(checkpoint_path, 'w') as f:
-        json.dump({"pageid": full_rows[middle_idx]["articleid"], "revid": resume_revid}, f)
-
     try:
         tester_partial.call_wikiq("--fandom-2020", "--resume")
     except subprocess.CalledProcessError as exc:
@@ -73,6 +66,7 @@ def test_resume():
     assert_frame_equal(df_full, df_resumed)
 
 
+@requires_pywikidiff2
 def test_resume_with_diff():
     """Test that --resume correctly computes diff values after resume.
 
@@ -103,10 +97,6 @@ def test_resume_with_diff():
         for row in full_rows[:resume_idx + 1]:
             f.write(json.dumps(row) + "\n")
 
-    checkpoint_path = get_checkpoint_path(partial_output_path)
-    with open(checkpoint_path, 'w') as f:
-        json.dump({"pageid": full_rows[resume_idx]["articleid"], "revid": resume_revid}, f)
-
     try:
         tester_partial.call_wikiq("--diff", "--fandom-2020", "--resume")
     except subprocess.CalledProcessError as exc:
@@ -170,10 +160,6 @@ def test_resume_simple():
         for row in full_rows[:resume_idx + 1]:
             f.write(json.dumps(row) + "\n")
 
-    checkpoint_path = get_checkpoint_path(partial_output_path)
-    with open(checkpoint_path, 'w') as f:
-        json.dump({"pageid": full_rows[resume_idx]["articleid"], "revid": resume_revid}, f)
-
     try:
         tester_partial.call_wikiq("--resume")
     except subprocess.CalledProcessError as exc:
@@ -186,39 +172,6 @@ def test_resume_simple():
     assert_frame_equal(df_full, df_resumed)
 
 
-def test_checkpoint_read():
-    """Test that read_checkpoint correctly reads checkpoint files."""
-    with tempfile.TemporaryDirectory() as tmpdir:
-        checkpoint_path = os.path.join(tmpdir, "test.jsonl.checkpoint")
-
-        # Test reading valid checkpoint
-        with open(checkpoint_path, 'w') as f:
-            json.dump({"pageid": 100, "revid": 200}, f)
-
-        result = read_checkpoint(checkpoint_path)
-        assert result == (100, 200), f"Expected (100, 200), got {result}"
-
-        # Test reading non-existent checkpoint
-        result = read_checkpoint(os.path.join(tmpdir, "nonexistent.checkpoint"))
-        assert result is None, f"Expected None for non-existent file, got {result}"
-
-        # Test reading empty checkpoint
-        empty_path = os.path.join(tmpdir, "empty.checkpoint")
-        with open(empty_path, 'w') as f:
-            f.write("{}")
-        result = read_checkpoint(empty_path)
-        assert result is None, f"Expected None for empty checkpoint, got {result}"
-
-        # Test reading corrupted checkpoint
-        corrupt_path = os.path.join(tmpdir, "corrupt.checkpoint")
-        with open(corrupt_path, 'w') as f:
-            f.write("not valid json")
-        result = read_checkpoint(corrupt_path)
-        assert result is None, f"Expected None for corrupted checkpoint, got {result}"
-
-    print("Checkpoint read test passed!")
-
-
 def test_resume_with_interruption():
     """Test that resume works correctly after interruption."""
     import pandas as pd
@@ -245,9 +198,6 @@ def test_resume_with_interruption():
     # Clean up for interrupted run
     if os.path.exists(output_file):
         os.remove(output_file)
-    checkpoint_path = get_checkpoint_path(output_file)
-    if os.path.exists(checkpoint_path):
-        os.remove(checkpoint_path)
 
     # Start wikiq and interrupt it
     cmd_partial = [
@@ -301,48 +251,6 @@ def test_resume_with_interruption():
     assert_frame_equal(df_full, df_resumed)
 
 
-def test_resume_parquet():
-    """Test that --resume works correctly with Parquet output format."""
-    import pandas as pd
-    from pandas.testing import assert_frame_equal
-    import pyarrow.parquet as pq
-
-    tester_full = WikiqTester(SAILORMOON, "resume_parquet_full", in_compression="7z", out_format="parquet")
-
-    try:
-        tester_full.call_wikiq("--fandom-2020")
-    except subprocess.CalledProcessError as exc:
-        pytest.fail(exc.stderr.decode("utf8"))
-
-    full_output_path = tester_full.output
-    full_table = pq.read_table(full_output_path)
-
-    # Use unsorted indices consistently - slice the table and get checkpoint from same position
-    resume_idx = len(full_table) // 3
-    resume_revid = int(full_table.column("revid")[resume_idx].as_py())
-    resume_pageid = int(full_table.column("articleid")[resume_idx].as_py())
-
-    tester_partial = WikiqTester(SAILORMOON, "resume_parquet_partial", in_compression="7z", out_format="parquet")
-    partial_output_path = tester_partial.output
-
-    # Write partial Parquet file using the SAME schema as the full file
-    partial_table = full_table.slice(0, resume_idx + 1)
-    pq.write_table(partial_table, partial_output_path)
-
-    checkpoint_path = get_checkpoint_path(partial_output_path)
-    with open(checkpoint_path, 'w') as f:
-        json.dump({"pageid": resume_pageid, "revid": resume_revid}, f)
-
-    try:
-        tester_partial.call_wikiq("--fandom-2020", "--resume")
-    except subprocess.CalledProcessError as exc:
-        pytest.fail(exc.stderr.decode("utf8"))
-
-    df_full = full_table.to_pandas()
-    df_resumed = pd.read_parquet(partial_output_path)
-    assert_frame_equal(df_full, df_resumed)
-
-
 def test_resume_tsv_error():
     """Test that --resume with TSV output produces a proper error message."""
     tester = WikiqTester(SAILORMOON, "resume_tsv_error", in_compression="7z", out_format="tsv")
@@ -352,7 +260,7 @@ def test_resume_tsv_error():
         pytest.fail("Expected error for --resume with TSV output")
     except subprocess.CalledProcessError as exc:
         stderr = exc.stderr.decode("utf8")
-        assert "Error: --resume only works with JSONL or Parquet" in stderr, \
+        assert "Error: --resume only works with JSONL output" in stderr, \
             f"Expected proper error message, got: {stderr}"
 
     print("TSV resume error test passed!")
@@ -387,10 +295,6 @@ def test_resume_data_equivalence():
         for row in full_rows[:resume_idx + 1]:
             f.write(json.dumps(row) + "\n")
 
-    checkpoint_path = get_checkpoint_path(partial_output_path)
-    with open(checkpoint_path, 'w') as f:
-        json.dump({"pageid": full_rows[resume_idx]["articleid"], "revid": resume_revid}, f)
-
     try:
         tester_partial.call_wikiq("--fandom-2020", "--resume")
     except subprocess.CalledProcessError as exc:
@@ -403,6 +307,7 @@ def test_resume_data_equivalence():
     assert_frame_equal(df_full, df_resumed)
 
 
+@requires_pywikidiff2
 def test_resume_with_persistence():
     """Test that --resume correctly handles persistence state after resume.
 
@@ -433,10 +338,6 @@ def test_resume_with_persistence():
         for row in full_rows[:resume_idx + 1]:
             f.write(json.dumps(row) + "\n")
 
-    checkpoint_path = get_checkpoint_path(partial_output_path)
-    with open(checkpoint_path, 'w') as f:
-        json.dump({"pageid": full_rows[resume_idx]["articleid"], "revid": resume_revid}, f)
-
     try:
         tester_partial.call_wikiq("--persistence wikidiff2", "--fandom-2020", "--resume")
     except subprocess.CalledProcessError as exc:
@@ -487,10 +388,6 @@ def test_resume_corrupted_jsonl_last_line():
     # Record file size before resume
     size_before = os.path.getsize(corrupt_output_path)
 
-    # NO checkpoint file - JSONL resume works from last valid line in the file
-    checkpoint_path = get_checkpoint_path(corrupt_output_path)
-    assert not os.path.exists(checkpoint_path), "Test setup error: checkpoint should not exist"
-
     # Resume should detect corrupted line, truncate it, then append new data
     try:
         tester_corrupt.call_wikiq("--fandom-2020", "--resume")
@@ -509,6 +406,7 @@ def test_resume_corrupted_jsonl_last_line():
     assert_frame_equal(df_full, df_resumed)
 
 
+@requires_pywikidiff2
 def test_resume_diff_persistence_combined():
     """Test that --resume correctly handles both diff and persistence state together.
 
@@ -537,10 +435,6 @@ def test_resume_diff_persistence_combined():
         for row in full_rows[:resume_idx + 1]:
             f.write(json.dumps(row) + "\n")
 
-    checkpoint_path = get_checkpoint_path(partial_output_path)
-    with open(checkpoint_path, 'w') as f:
-        json.dump({"pageid": full_rows[resume_idx]["articleid"], "revid": resume_revid}, f)
-
     try:
         tester_partial.call_wikiq("--diff", "--persistence wikidiff2", "--fandom-2020", "--resume")
     except subprocess.CalledProcessError as exc:
@@ -558,6 +452,7 @@ def test_resume_diff_persistence_combined():
     assert_frame_equal(df_full, df_resumed)
 
 
+@requires_pywikidiff2
 def test_resume_mid_page():
     """Test resume from the middle of a page with many revisions.
 
@@ -588,7 +483,7 @@ def test_resume_mid_page():
     resume_revid = int(resume_rev["revid"])
     resume_pageid = int(resume_rev["articleid"])
 
-    # Find global index for checkpoint
+    # Find global index for the resume point
     global_idx = df_full[df_full["revid"] == resume_revid].index[0]
 
     tester_partial = WikiqTester(SAILORMOON, "resume_midpage_partial", in_compression="7z", out_format="jsonl")
@@ -600,10 +495,6 @@ def test_resume_mid_page():
         for row in rows_to_write:
             f.write(json.dumps(row) + "\n")
 
-    checkpoint_path = get_checkpoint_path(partial_output_path)
-    with open(checkpoint_path, 'w') as f:
-        json.dump({"pageid": resume_pageid, "revid": resume_revid}, f)
-
     try:
         tester_partial.call_wikiq("--diff", "--fandom-2020", "--resume")
     except subprocess.CalledProcessError as exc:
@@ -615,6 +506,7 @@ def test_resume_mid_page():
     assert_frame_equal(df_full, df_resumed)
 
 
+@requires_pywikidiff2
 def test_resume_page_boundary():
     """Test resume at the exact start of a new page.
 
@@ -653,10 +545,6 @@ def test_resume_page_boundary():
         for row in rows_to_write:
             f.write(json.dumps(row) + "\n")
 
-    checkpoint_path = get_checkpoint_path(partial_output_path)
-    with open(checkpoint_path, 'w') as f:
-        json.dump({"pageid": resume_pageid, "revid": resume_revid}, f)
-
     try:
         tester_partial.call_wikiq("--diff", "--fandom-2020", "--resume")
     except subprocess.CalledProcessError as exc:
@@ -790,10 +678,6 @@ def test_resume_revert_detection():
         for row in full_rows[:resume_idx + 1]:
             f.write(json.dumps(row) + "\n")
 
-    checkpoint_path = get_checkpoint_path(partial_output_path)
-    with open(checkpoint_path, 'w') as f:
-        json.dump({"pageid": resume_pageid, "revid": resume_revid}, f)
-
     try:
         tester_partial.call_wikiq("--fandom-2020", "--resume")
     except subprocess.CalledProcessError as exc:
@@ -806,3 +690,40 @@ def test_resume_revert_detection():
     # Verify revert column matches exactly
     assert_series_equal(df_full["revert"], df_resumed["revert"])
     assert_series_equal(df_full["reverteds"], df_resumed["reverteds"])
+
+
+def test_resume_collapse_user():
+    """Test that --resume with --collapse-user matches an uninterrupted run."""
+    import pandas as pd
+    from pandas.testing import assert_frame_equal
+
+    tester_full = WikiqTester(SAILORMOON, "resume_collapse_full", in_compression="7z", out_format="jsonl")
+
+    try:
+        tester_full.call_wikiq("--collapse-user", "--fandom-2020")
+    except subprocess.CalledProcessError as exc:
+        pytest.fail(exc.stderr.decode("utf8"))
+
+    full_rows = read_jsonl(tester_full.output)
+
+    # Truncate in the middle of a page with several collapsed rows so the
+    # resume point falls between collapsed revision groups
+    middle_idx = len(full_rows) // 2
+
+    tester_partial = WikiqTester(SAILORMOON, "resume_collapse_partial", in_compression="7z", out_format="jsonl")
+    partial_output_path = tester_partial.output
+
+    with open(partial_output_path, 'w') as f:
+        for row in full_rows[:middle_idx + 1]:
+            f.write(json.dumps(row) + "\n")
+
+    try:
+        tester_partial.call_wikiq("--collapse-user", "--fandom-2020", "--resume")
+    except subprocess.CalledProcessError as exc:
+        pytest.fail(exc.stderr.decode("utf8"))
+
+    resumed_rows = read_jsonl(partial_output_path)
+
+    df_full = pd.DataFrame(full_rows)
+    df_resumed = pd.DataFrame(resumed_rows)
+    assert_frame_equal(df_full, df_resumed)

test/test_wiki_diff_matcher.py

@@ -7,6 +7,10 @@ from typing import List
 from deltas import Delete, Equal, Insert, wikitext_split
 from mwpersistence import Token
 from wikiq.wiki_diff_matcher import WikiDiffMatcher
+from wikiq_test_utils import requires_pywikidiff2
+
+# every test here drives wikidiff2 directly
+pytestmark = requires_pywikidiff2
 
 def _replace_whitespace(match):
     if match.group(1):  # If spaces matched (e.g., '  ')
@@ -21,8 +25,6 @@ def assert_equal_enough(tokens:List[Token], rev):
     # the tokens exclude newlines
     # we allow extra whitespace at the beginning or end
     token_doc = ''.join(str(t) for t in tokens)
-    print(token_doc, file = open('token','w'))
-    print(rev, file = open('rev','w'))
     token_doc = re.sub(r'( +)|(\n+)|(\t+)', _replace_whitespace, token_doc).strip()
     rev = re.sub(r'( +)|(\n+)|(\t+)', _replace_whitespace, rev).strip()
     assert token_doc == rev
@@ -349,6 +351,10 @@ def test_actually_equal():
     assert_equal_enough(a, rev1)
     
 # slow test. comment out the following line to enable it.
+# Requires an uncompressed test/dumps/ikwiki.xml, which is not in the
+# repository; decompress the .bz2 in test/dumps first. Also writes debug
+# output to test_unicode_highlight_from/_to in the current directory on
+# every iteration.
 @pytest.mark.skip
 def test_diff_consistency():
     from mwxml import Dump
@@ -366,6 +372,9 @@ def test_diff_consistency():
             assert_equal_enough(b, rev)
             last_rev = rev
 
+# benchmark, not a pass/fail test; run it deliberately when tuning diff
+# performance. Requires an uncompressed test/dumps/ikwiki.xml, which is
+# not in the repository; decompress the .bz2 in test/dumps first.
 @pytest.mark.skip
 def test_benchmark_diff(benchmark):
     from mwxml import Dump

test/test_wikiq.py

@@ -12,7 +12,7 @@ import pytest
 from pandas import DataFrame
 from pandas.testing import assert_frame_equal, assert_series_equal
 
-from wikiq import build_table, build_schema
+from wikiq import build_table, build_schema, RegexPair, WikitextParser
 from wikiq_test_utils import (
     BASELINE_DIR,
     IKWIKI,
@@ -23,6 +23,8 @@ from wikiq_test_utils import (
     TWINPEAKS,
     WIKIQ,
     WikiqTester,
+    requires_mediawiki_utilities,
+    requires_pywikidiff2,
 )
 
 
@@ -38,7 +40,7 @@ setup()
 
 def read_jsonl_with_schema(filepath: str, **schema_kwargs) -> pd.DataFrame:
     """Read JSONL file using PyArrow with explicit schema from wikiq."""
-    table, _ = build_table(**schema_kwargs)
+    table, _, _ = build_table(**schema_kwargs)
     schema = build_schema(table, **schema_kwargs)
     pa_table = pj.read_json(
         filepath,
@@ -205,6 +207,7 @@ def test_collapse_user():
     baseline = pd.read_table(tester.baseline_file)
     assert_frame_equal(test, baseline, check_like=True)
 
+@requires_pywikidiff2
 def test_pwr_wikidiff2():
     tester = WikiqTester(SAILORMOON, "persistence_wikidiff2", in_compression="7z")
 
@@ -229,6 +232,7 @@ def test_pwr_segment():
     baseline = pd.read_table(tester.baseline_file)
     assert_frame_equal(test, baseline, check_like=True)
 
+@requires_mediawiki_utilities
 def test_pwr_legacy():
     tester = WikiqTester(SAILORMOON, "persistence_legacy", in_compression="7z")
 
@@ -255,6 +259,7 @@ def test_pwr():
     test = test.reindex(columns=sorted(test.columns))
     assert_frame_equal(test, baseline, check_like=True)
 
+@requires_pywikidiff2
 def test_diff():
     tester = WikiqTester(SAILORMOON, "diff", in_compression="7z", out_format='jsonl')
 
@@ -268,6 +273,7 @@ def test_diff():
     assert "diff_timeout" in test.columns, "diff_timeout column should exist"
     assert len(test) > 0, "Should have output rows"
 
+@requires_pywikidiff2
 def test_diff_plus_pwr():
     tester = WikiqTester(SAILORMOON, "diff_pwr", in_compression="7z", out_format='jsonl')
 
@@ -281,6 +287,7 @@ def test_diff_plus_pwr():
     assert "token_revs" in test.columns, "token_revs column should exist"
     assert len(test) > 0, "Should have output rows"
 
+@requires_pywikidiff2
 def test_text():
     tester = WikiqTester(SAILORMOON, "text", in_compression="7z", out_format='jsonl')
 
@@ -391,6 +398,157 @@ def test_capturegroup_regex():
         baseline = pd.read_table(tester.baseline_file)
         assert_frame_equal(test, baseline, check_like=True)
 
+def test_regex_none_content():
+    # deleted or suppressed revisions yield None for text and comments, and
+    # matchmake must tolerate that in both the capture-group and plain paths
+    pair = RegexPair(r"(?P<letter>\b[a-zA-Z]{3}\b)|(?P<number>\b\d+\b)", "cap")
+    assert pair.matchmake(None) == {"cap_letter": None, "cap_number": None}
+
+    pair = RegexPair(r"\b\d{3}\b", "digits")
+    assert pair.matchmake(None) == {"digits": None}
+
+def test_regex_deleted_revisions():
+    # the ikwiki dump contains revisions with deleted text and deleted
+    # comments; regex matching must handle them rather than crashing
+    tester = WikiqTester(wiki=IKWIKI, case_name="regex_deleted")
+
+    try:
+        tester.call_wikiq(
+            "-RP '(?P<npov>npov|NPOV)' -RPl npov",
+            "-CP '(?P<talk>[Tt]alk)' -CPl talk",
+        )
+    except subprocess.CalledProcessError as exc:
+        pytest.fail(exc.stderr.decode("utf8"))
+
+    test = pd.read_table(tester.output)
+
+    deleted = test[test["deleted"]]
+    assert len(deleted) > 0
+    assert deleted["npov_npov"].isna().all()
+    assert deleted["talk_talk"].isna().all()
+
+def test_redirect_detection():
+    from wikiq.tables import RedirectDetector
+
+    detector = RedirectDetector()
+
+    # a plain redirect directive
+    assert detector.detect("#REDIRECT [[Target]]") == (True, "Target")
+
+    # localized aliases are only recognized when configured
+    assert detector.detect("#OMDIRIGERING [[Mål]]") == (False, None)
+    swedish = RedirectDetector(["OMDIRIGERING"])
+    assert swedish.detect("#OMDIRIGERING [[Mål]]") == (True, "Mål")
+    assert swedish.detect("#REDIRECT [[Target]]") == (True, "Target")
+
+    # fragment and label are stripped from the target
+    assert detector.detect("#REDIRECT [[Target#Section|label]]") == (True, "Target")
+
+    # leading whitespace and lowercase are accepted
+    assert detector.detect("  \n#redirect [[Target]]") == (True, "Target")
+
+    # each revision is classified by its own text
+    history = ["#REDIRECT [[A]]", "An article now.", "#REDIRECT [[B]]"]
+    assert [detector.detect(t) for t in history] == [
+        (True, "A"), (False, None), (True, "B"),
+    ]
+
+    # a directive that is not at the start of the text is not a redirect
+    assert detector.detect("Some text. #REDIRECT [[Target]]") == (False, None)
+
+def test_redirect_columns_e2e():
+    # revision-level redirect columns on a real dump: the ikwiki dump
+    # contains both redirect revisions and revisions with deleted text
+    tester = WikiqTester(IKWIKI, "redirect_columns")
+
+    try:
+        tester.call_wikiq()
+    except subprocess.CalledProcessError as exc:
+        pytest.fail(exc.stderr.decode("utf8"))
+
+    test = pd.read_table(tester.output)
+
+    assert "revision_is_redirect" in test.columns
+    assert "revision_redirect_target" in test.columns
+    assert "redirect_target" not in test.columns
+
+    # revisions with deleted text have null values in both columns
+    deleted = test[test["deleted"]]
+    assert len(deleted) > 0
+    assert deleted["revision_is_redirect"].isna().all()
+    assert deleted["revision_redirect_target"].isna().all()
+
+    # the dump contains real redirect revisions, and every detected
+    # redirect has a target
+    redirects = test[test["revision_is_redirect"] == True]
+    assert len(redirects) > 0
+    assert redirects["revision_redirect_target"].notna().all()
+
+def test_content_sizes():
+    # hand-computed fixtures: (text, content_chars, markup_chars)
+    cases = [
+        # plain text is all content
+        ("Hello world.", 12, 0),
+        # the same information written by someone who knows wikitext: the
+        # values survive as content, the scaffolding around them is markup
+        ("Wonderful Hotel, +15555550123", 29, 0),
+        ("{{listing|name=Wonderful Hotel|phone=+15555550123}}", 27, 24),
+        # emphasis is markup applied to content, not content itself
+        ("''whatever''", 8, 4),
+        ("Some '''bold''' prose.", 16, 6),
+        # tag contents are content; the tag name and attributes are not
+        ('Before <span class="x">inner text</span> after', 23, 23),
+        # a wikilink renders its label, so the target is markup
+        ("[[Main Page|the main page]]", 13, 14),
+        # an unlabeled wikilink renders its target, which is then content
+        ("[[Help]]", 4, 4),
+        # both halves of an external link are information someone typed
+        ("[http://x.com label]", 17, 3),
+        # comments never reach a reader, so they are entirely markup
+        ("Text <!-- hidden --> more", 10, 15),
+        # nesting: values at any depth are content
+        ("{{outer|a={{inner|b=deep}} shallow}}", 12, 24),
+    ]
+
+    parser = WikitextParser()
+    for text, content, markup in cases:
+        result = parser.content_sizes(text)
+        assert result == (content, markup), f"{text!r}: got {result}, want {(content, markup)}"
+        # every character lands in exactly one column
+        assert result[0] + result[1] == len(text), f"{text!r}: columns do not sum to length"
+
+    assert parser.content_sizes(None) is None
+
+def test_content_sizes_e2e():
+    # the ikwiki dump contains revisions with deleted text, which must get
+    # null content sizes; for every other revision the two columns must
+    # account for the whole revision
+    tester = WikiqTester(IKWIKI, "content_sizes")
+
+    try:
+        tester.call_wikiq("--content-sizes", "--text")
+    except subprocess.CalledProcessError as exc:
+        pytest.fail(exc.stderr.decode("utf8"))
+
+    test = pd.read_table(tester.output)
+
+    assert "content_chars" in test.columns
+    assert "markup_chars" in test.columns
+
+    deleted = test[test["deleted"]]
+    assert len(deleted) > 0
+    assert deleted["content_chars"].isna().all()
+    assert deleted["markup_chars"].isna().all()
+
+    present = test[~test["deleted"] & test["text"].notna() & test["content_chars"].notna()]
+    assert len(present) > 0
+    total = present["content_chars"] + present["markup_chars"]
+    assert (total == present["text"].str.len()).all()
+    # a real wiki has both kinds of revision: some almost pure text, some
+    # carrying a lot of markup
+    assert (present["markup_chars"] == 0).any()
+    assert (present["markup_chars"] > 0).any()
+
 def test_external_links_only():
     """Test that --external-links extracts external links correctly."""
     import mwparserfromhell
@@ -411,7 +569,7 @@ def test_external_links_only():
     # Verify citations column does NOT exist
     assert "citations" not in test.columns, "citations column should NOT exist when only --external-links is used"
 
-    # Verify column has list/array type (pandas reads parquet lists as numpy arrays)
+    # Verify column has list/array type (pandas reads pyarrow lists as numpy arrays)
     assert test["external_links"].apply(lambda x: x is None or hasattr(x, '__len__')).all(), \
         "external_links should be a list/array type or None"
 
@@ -466,7 +624,7 @@ def test_citations_only():
     # Verify external_links column does NOT exist
     assert "external_links" not in test.columns, "external_links column should NOT exist when only --citations is used"
 
-    # Verify column has list/array type (pandas reads parquet lists as numpy arrays)
+    # Verify column has list/array type (pandas reads pyarrow lists as numpy arrays)
     assert test["citations"].apply(lambda x: x is None or hasattr(x, '__len__')).all(), \
         "citations should be a list/array type or None"
 
@@ -516,7 +674,7 @@ def test_external_links_and_citations():
     assert "external_links" in test.columns, "external_links column should exist"
     assert "citations" in test.columns, "citations column should exist"
 
-    # Verify both columns have list/array types (pandas reads parquet lists as numpy arrays)
+    # Verify both columns have list/array types (pandas reads pyarrow lists as numpy arrays)
     assert test["external_links"].apply(lambda x: x is None or hasattr(x, '__len__')).all(), \
         "external_links should be a list/array type or None"
     assert test["citations"].apply(lambda x: x is None or hasattr(x, '__len__')).all(), \
@@ -724,37 +882,4 @@ def test_headings():
     print(f"Headings test passed! {len(test)} rows processed")
 
 
-def test_parquet_output():
-    """Test that Parquet output format works correctly."""
-    tester = WikiqTester(SAILORMOON, "parquet_output", in_compression="7z", out_format="parquet")
-
-    try:
-        tester.call_wikiq("--fandom-2020")
-    except subprocess.CalledProcessError as exc:
-        pytest.fail(exc.stderr.decode("utf8"))
-
-    # Verify output file exists
-    assert os.path.exists(tester.output), f"Parquet output file should exist at {tester.output}"
-
-    # Read and verify content
-    test = pd.read_parquet(tester.output)
-
-    # Verify expected columns exist
-    assert "revid" in test.columns
-    assert "articleid" in test.columns
-    assert "title" in test.columns
-    assert "namespace" in test.columns
-
-    # Verify row count matches JSONL output
-    tester_jsonl = WikiqTester(SAILORMOON, "parquet_compare", in_compression="7z", out_format="jsonl")
-    try:
-        tester_jsonl.call_wikiq("--fandom-2020")
-    except subprocess.CalledProcessError as exc:
-        pytest.fail(exc.stderr.decode("utf8"))
-
-    test_jsonl = pd.read_json(tester_jsonl.output, lines=True)
-    assert len(test) == len(test_jsonl), f"Parquet and JSONL should have same row count: {len(test)} vs {len(test_jsonl)}"
-
-    print(f"Parquet output test passed! {len(test)} rows")
-
 

test/wikiq_test_utils.py

@@ -1,8 +1,27 @@
+import importlib.util
 import os
 import shutil
 import subprocess
 from typing import Final, Union
 
+import pytest
+
+
+def _installed(module: str) -> bool:
+    return importlib.util.find_spec(module) is not None
+
+
+# wikiq's two optional dependencies. Tests exercising --diff, -p wikidiff2, or
+# -p legacy cannot run on a base install, so they skip rather than fail.
+requires_pywikidiff2 = pytest.mark.skipif(
+    not _installed("pywikidiff2"),
+    reason="needs the optional pywikidiff2 extension (--diff, -p wikidiff2)",
+)
+requires_mediawiki_utilities = pytest.mark.skipif(
+    not _installed("mw"),
+    reason="needs the optional mediawiki-utilities package (-p legacy)",
+)
+
 TEST_DIR: Final[str] = os.path.dirname(os.path.realpath(__file__))
 WIKIQ: Final[str] = os.path.join(os.path.join(TEST_DIR, ".."), "src/wikiq/__init__.py")
 TEST_OUTPUT_DIR: Final[str] = os.path.join(TEST_DIR, "test_output")
@@ -42,17 +61,8 @@ class WikiqTester:
             else:
                 shutil.rmtree(self.output)
 
-        # Also clean up resume-related files
-        for temp_suffix in [".resume_temp", ".checkpoint", ".merged"]:
-            temp_path = self.output + temp_suffix
-            if os.path.exists(temp_path):
-                if os.path.isfile(temp_path):
-                    os.remove(temp_path)
-                else:
-                    shutil.rmtree(temp_path)
-
-        # For JSONL and Parquet, self.output is a file path. Create parent directory if needed.
-        if out_format in ("jsonl", "parquet"):
+        # For JSONL, self.output is a file path. Create parent directory if needed.
+        if out_format == "jsonl":
             parent_dir = os.path.dirname(self.output)
             if parent_dir:
                 os.makedirs(parent_dir, exist_ok=True)