Commit Graph

Select branches

Hide Pull Requests

content-sizes

jsonl-output

mako_changes-20260806

master

redirects

regex-counting

release_review

#1

#2

#3

#4

archive/content-sizes-preconsolidation

archive/legacy

archive/regex_scanner

archive/tests

archive/user_level_wikiq

v0.1.1

Mono Color

• af62c4cea4 add --content-sizes: split revisions into content and markup content-sizes Benjamin Mako Hill 2026-08-06 18:09:29 -07:00
• e4c05caf94 document regex match counting in the README regex-counting Benjamin Mako Hill 2026-08-13 15:51:11 -07:00
• 8056c91ac8 add regex match counting (-RPc / -CPc) Benjamin Mako Hill 2026-08-06 15:47:38 -07:00
• 57988849cc commit the uv lockfile release_review Benjamin Mako Hill 2026-08-13 17:13:14 -07:00
• 268c212469 sort the imports in wikiq/__init__.py Benjamin Mako Hill 2026-08-13 16:53:52 -07:00
• 8bd6c2e129 add a CHANGELOG Benjamin Mako Hill 2026-08-13 16:16:22 -07:00
• 0fa8b3d9ba put the base install ahead of the optional features in the README Benjamin Mako Hill 2026-08-13 16:15:55 -07:00
• 1b55acb01f rename the project from mediawiki_dump_tools to wikiq Benjamin Mako Hill 2026-08-13 16:15:14 -07:00
• b073b038af add --version and set the version to 0.3.0 Benjamin Mako Hill 2026-08-06 16:33:55 -07:00
• 77f185888f run the whole suite from runtest.sh Benjamin Mako Hill 2026-08-13 13:28:06 -07:00
• ff1f42dae5 document the optional dependencies Benjamin Mako Hill 2026-08-13 12:00:42 -07:00
• 507cc24328 correct the test suite runtime in the README Benjamin Mako Hill 2026-08-08 11:45:04 -07:00
• 1125194093 add the project metadata a published package needs Benjamin Mako Hill 2026-08-13 13:24:44 -07:00
• 421e9c0739 keep local files out of the source distribution Benjamin Mako Hill 2026-08-13 12:00:42 -07:00
• a68a596bf4 remove the stale .gitmodules Benjamin Mako Hill 2026-08-13 13:20:59 -07:00
• fbaafd1381 ignore the venv and local tool state Benjamin Mako Hill 2026-08-08 11:39:14 -07:00
• af8c45e8c0 rename Wikiq_Unit_Test.py to test_wikiq.py Benjamin Mako Hill 2026-08-13 11:59:57 -07:00
• b8dddf6af8 make redirect detection revision-level Benjamin Mako Hill 2026-08-06 23:41:57 -07:00
• d4111dc4f3 update README for the current tool Benjamin Mako Hill 2026-08-06 16:30:21 -07:00
• ceb230904e license under the GNU GPL, version 3 or later Benjamin Mako Hill 2026-08-06 16:29:48 -07:00
• c31833e064 convert README to markdown Benjamin Mako Hill 2026-08-06 16:23:52 -07:00
• 15379c1c1a require Python 3.11 Benjamin Mako Hill 2026-08-13 13:35:14 -07:00
• ac8110894b resolve dependencies from PyPI instead of pinned git forks Benjamin Mako Hill 2026-08-13 13:35:14 -07:00
• b10a7d0130 skip tests whose optional dependency is not installed Benjamin Mako Hill 2026-08-13 11:58:30 -07:00
• 70b0ebcf0c make mediawiki-utilities an optional dependency Benjamin Mako Hill 2026-08-13 11:58:12 -07:00
• 5725c66040 make pywikidiff2 an optional dependency Benjamin Mako Hill 2026-08-13 11:57:36 -07:00
• 6013e8c4e9 remove pyspark dependency and wikiq_spark packaging references Benjamin Mako Hill 2026-08-06 15:50:50 -07:00
• 015d4f9164 remove parquet output support Benjamin Mako Hill 2026-08-06 15:31:48 -07:00
• caf604e43d document the requirements of the two skipped diff matcher tests Benjamin Mako Hill 2026-08-08 14:52:59 -07:00
• 04ccea4848 remove orphaned regextest.tsv baseline Benjamin Mako Hill 2026-08-06 16:23:52 -07:00
• f4f0ed72a0 stop writing debug files from the diff matcher test helper Benjamin Mako Hill 2026-08-06 16:22:39 -07:00
• 7fbf62e5e0 add test for --resume with --collapse-user Benjamin Mako Hill 2026-08-06 15:47:28 -07:00
• 1b3093dec0 regenerate test baselines for the redirect_target column Benjamin Mako Hill 2026-08-06 15:34:38 -07:00
• a26abe0e68 fix test helper for build_table's three-value return Benjamin Mako Hill 2026-08-06 15:33:34 -07:00
• 97e21e3634 only prefix namespace names onto titles that lack them Benjamin Mako Hill 2026-08-06 16:05:44 -07:00
• c61eb10e47 fix regex matching on revisions with deleted text or comments Benjamin Mako Hill 2026-08-06 14:20:02 -07:00
• c49fa5af3c Merge branch 'mako_changes-20260806' into content-sizes archive/content-sizes-preconsolidation Benjamin Mako Hill 2026-08-13 08:52:15 -07:00
• dfd2dbe20f resolve yamlconf from PyPI mako_changes-20260806 Benjamin Mako Hill 2026-08-13 08:40:06 -07:00
• 851f070c8f Merge branch 'mako_changes-20260806' into content-sizes Benjamin Mako Hill 2026-08-08 14:55:06 -07:00
• fde1452666 document the requirements of the two skipped diff matcher tests Benjamin Mako Hill 2026-08-08 14:52:59 -07:00
• c6117a7182 Merge remote-tracking branch 'gitea/jsonl-output' into mako_changes-20260806 Benjamin Mako Hill 2026-08-08 14:42:49 -07:00
• 0b1557efc5 note that installs are copies and point CDSC users to the wiki Benjamin Mako Hill 2026-08-08 12:06:16 -07:00
• aedd4b3086 Merge branch 'mako_changes-20260806' into content-sizes Benjamin Mako Hill 2026-08-08 12:01:20 -07:00
• 2bee25488b correct the test suite runtime in the README Benjamin Mako Hill 2026-08-08 11:45:04 -07:00
• 702101bfe2 ignore the .venv directory Benjamin Mako Hill 2026-08-08 11:39:14 -07:00
• 41fda639d8 add infobox test fixtures jsonl-output Nathan TeBlunthuis 2026-08-07 22:27:10 -05:00
• 1700561b07 Merge branch 'mako_changes-20260806' into content-sizes Benjamin Mako Hill 2026-08-06 23:52:31 -07:00
• 40132f04dd make redirect detection revision-level Benjamin Mako Hill 2026-08-06 23:41:57 -07:00
• 3d6bf34c8e add --content-sizes: decompose revision content into prose and structured characters Benjamin Mako Hill 2026-08-06 18:09:29 -07:00
• c1ee926211 add --version and bump to 0.2.0 Benjamin Mako Hill 2026-08-06 16:33:55 -07:00
• d6d0aab604 update README for the current tool Benjamin Mako Hill 2026-08-06 16:30:21 -07:00
• 90a14cf999 license under the GNU GPL, version 3 or later Benjamin Mako Hill 2026-08-06 16:29:48 -07:00
• 5fb7b5596c remove orphaned regextest.tsv baseline Benjamin Mako Hill 2026-08-06 16:23:52 -07:00
• fa8adc30ca convert README to markdown Benjamin Mako Hill 2026-08-06 16:23:52 -07:00
• 45f0680200 stop writing debug files from the diff matcher test helper Benjamin Mako Hill 2026-08-06 16:22:39 -07:00
• 0ed0f57dbb require Python 3.11 Benjamin Mako Hill 2026-08-06 16:17:14 -07:00
• 14886106ed resolve dependencies from PyPI instead of pinned github forks Benjamin Mako Hill 2026-08-06 16:05:44 -07:00
• ec09641545 only prefix namespace names onto titles that lack them Benjamin Mako Hill 2026-08-06 16:05:44 -07:00
• 3ab988d6eb declare pywikidiff2 as a PEP 508 direct reference over https Benjamin Mako Hill 2026-08-06 16:00:44 -07:00
• 392211af17 remove pyspark dependency and wikiq_spark packaging references Benjamin Mako Hill 2026-08-06 15:50:50 -07:00
• 0b6e0fbc70 add regex match counting (-RPc / -CPc) Benjamin Mako Hill 2026-08-06 15:47:38 -07:00
• 6634de51e6 add test for --resume with --collapse-user Benjamin Mako Hill 2026-08-06 15:47:28 -07:00
• 83b7fc0210 skip test_infobox when its fixture files are absent Benjamin Mako Hill 2026-08-06 15:34:48 -07:00
• 791be0aa56 regenerate test baselines for the redirect_target column Benjamin Mako Hill 2026-08-06 15:34:38 -07:00
• 7c27942cb0 fix test helper for build_table's three-value return Benjamin Mako Hill 2026-08-06 15:33:34 -07:00
• 72a8851373 remove parquet output support Benjamin Mako Hill 2026-08-06 15:31:48 -07:00
• dbcae5c64e fix regex matching on revisions with deleted text or comments Benjamin Mako Hill 2026-08-06 14:20:02 -07:00
• 72410d090b wikiq: skip parse on subsequent same-size revs after timeout v0.1.1 Nathan TeBlunthuis 2026-05-07 21:43:05 +00:00
• 59fea1919e Add redirect_target column to wikiq output. Nathan TeBlunthuis 2026-03-01 20:05:12 -08:00
• c7eb374ceb use signalling to timeout mwparserfromhell instead of asyncio. Nathan TeBlunthuis 2026-01-07 12:42:37 -08:00
• 4b8288c016 add some debug lines. Nathan TeBlunthuis 2026-01-06 19:58:18 -08:00
• 8590e5f920 fix jsonl.d output. Nathan TeBlunthuis 2025-12-30 11:26:24 -08:00
• 93f6ed0ff5 fix bug by truncating corrupted jsonl lines. Nathan TeBlunthuis 2025-12-23 19:52:37 -08:00
• 5ebdb26d82 make resume with jsonl output fault tolerant. Nathan TeBlunthuis 2025-12-23 09:09:51 -08:00
• 9e6b0fb64c make updating the checkpoint files atomic. Nathan TeBlunthuis 2025-12-23 08:41:38 -08:00
• d822085698 support .jsonl.d Nathan TeBlunthuis 2025-12-22 20:13:04 -08:00
• 618c343898 allow output dir to be jsonl.d. Nathan TeBlunthuis 2025-12-21 23:50:00 -08:00
• 3f1a9ba862 refactor and enable jsonl output. Nathan TeBlunthuis 2025-12-21 23:42:18 -08:00
• 6988a281dc output parquet files in chunks to avoid memory issues with parquet. Nathan TeBlunthuis 2025-12-20 21:45:39 -08:00
• 6a4bf81e1a add test for two wikiq jobs in the same directory. Nathan TeBlunthuis 2025-12-19 11:50:56 -08:00
• 38dabd0547 only merge the correct partitioned files. Nathan TeBlunthuis 2025-12-19 11:47:18 -08:00
• 006feb795c fix interruption handling by breaking the diff loop. Nathan TeBlunthuis 2025-12-18 18:00:30 -08:00
• d7f5abef2d resume starts fresh if the first run didn't happen Nathan TeBlunthuis 2025-12-13 15:41:44 -08:00
• 2c54425726 use the wikidiff2 diff timeout instead of async. Nathan TeBlunthuis 2025-12-13 14:29:16 -08:00
• 5d1a246898 don't try to remove files that don't exist. Nathan TeBlunthuis 2025-12-13 11:57:47 -08:00
• 70a10db228 save work after a time limit. Nathan TeBlunthuis 2025-12-11 08:30:32 -08:00
• 1001c780fa start fresh if output and resume are both broken. Nathan TeBlunthuis 2025-12-10 21:20:52 -08:00
• 6b4f3939a5 more work on resuming. Nathan TeBlunthuis 2025-12-10 21:07:52 -08:00
• c3d31b4ab5 handle case when we have a valid resume file, but a corrupted original. Nathan TeBlunthuis 2025-12-10 20:33:04 -08:00
• f4a9491ff2 improve print debugging. Nathan TeBlunthuis 2025-12-10 19:50:47 -08:00
• c6e96c2f54 try/catch opening original file in resume. Nathan TeBlunthuis 2025-12-10 19:49:29 -08:00
• f427291fd8 add logic for resuming after a resume. Nathan TeBlunthuis 2025-12-10 19:26:54 -08:00
• d1fc094c96 don't put checkpoint files inside namespace directories. Nathan TeBlunthuis 2025-12-07 06:24:04 -08:00
• 783f5fd8bc improve resume logic. Nathan TeBlunthuis 2025-12-07 06:06:26 -08:00
• 577ddc87f5 Add per-namespace resume support for partitioned parquet output. Nathan TeBlunthuis 2025-12-06 06:56:19 -08:00
• d69d8b0df2 fix baseline output for new columns. Nathan TeBlunthuis 2025-12-02 19:22:08 -08:00
• 5ce9808b50 add templates and headings to wikiq. Nathan TeBlunthuis 2025-12-02 17:51:08 -08:00
• d3517ed5ca extract wikilinks. Nathan TeBlunthuis 2025-12-02 14:09:29 -08:00
• 329341efb6 improve tests. Nathan TeBlunthuis 2025-12-02 13:52:12 -08:00
• 76626a2785 Start working on adding columns from mwparserfromhell. Nathan TeBlunthuis 2025-12-02 12:26:03 -08:00