Commit Graph

Select branches

Hide Pull Requests

advanced_persistence

legacy

mako_changes-20230429

master

mediawiki-utils-migration

parquet_support

redirects

regex_scanner

tests

user_level_wikiq

#1

Mono Color

• 15e9234903 adding pyproject.toml parquet_support Nathan TeBlunthuis 2025-05-28 20:59:55 -07:00
• 8c7d46472f Merge branch 'parquet_support' of code:mediawiki_dump_tools into parquet_support Nathan TeBlunthuis 2025-05-28 20:54:52 -07:00
• 3c7fb088d6 fix schema bugs. Nathan TeBlunthuis 2025-05-28 20:54:42 -07:00
• 933ca753ed code review. mako_changes-20230429 Nathan TeBlunthuis 2023-05-03 10:23:30 -07:00
• 54fa6221a8 fix because pandas testing API has changed Benjamin Mako Hill 2023-04-29 11:52:13 -07:00
• 9dcd337315 rename variables to be more consistent Benjamin Mako Hill 2023-04-29 11:44:48 -07:00
• 2ff4d60613 added counting functionality to regex code Benjamin Mako Hill 2023-04-29 11:40:03 -07:00
• 4729371d5a updated README file Benjamin Mako Hill 2023-04-28 14:40:18 -07:00
• 7e6cd5b386 make sure that content is defined before testing for search patterns Benjamin Mako Hill 2023-04-28 14:30:42 -07:00
• 556285b198 added a line to fix persistence with deleted revs Benjamin Mako Hill 2023-04-28 14:21:21 -07:00
• b124f9c7c8 write regex captures to parquet arrays. redirects Nathan TeBlunthuis 2022-03-29 17:52:26 -07:00
• 32283aa4da add a minor comment on the source of the redirect regex Nathan TeBlunthuis 2022-03-10 15:07:27 -08:00
• 595728d8da resolve redirects if siteinfo is provided Nathan TeBlunthuis 2022-03-10 13:31:03 -08:00
• 3e645b5e58 bugfix. column name text_chars Nathan TeBlunthuis 2022-03-08 20:17:20 -08:00
• 1aea601a30 [Bugfix] Call the correct matchmake function. Nathan TeBlunthuis 2021-11-16 16:53:21 -08:00
• c437b357db rename matchmake functions Nathan TeBlunthuis 2021-11-11 19:09:41 -08:00
• bb83d62b74 Add some descriptive comments. Nathan TeBlunthuis 2021-10-19 16:55:24 -07:00
• c285402683 add todos to readme Nathan TeBlunthuis 2021-10-18 14:14:11 -07:00
• b1bea09ad6 fix bugs and unit tests Nathan TeBlunthuis 2021-10-18 13:33:05 -07:00
• 9a0c157ebb bugfix Nathan TeBlunthuis 2021-10-18 10:15:03 -07:00
• ae870fed0b parquet path is code-complete Nathan TeBlunthuis 2021-10-17 21:46:31 -07:00
• 26f6d8f984 remove dependency on pandas. Nathan TeBlunthuis 2021-10-17 20:24:33 -07:00
• ae9a241747 use dataclasses and pyarrow for types. Nathan TeBlunthuis 2021-10-17 20:21:22 -07:00
• d8d20f670b initial work on parquet support Nathan TeBlunthuis 2021-10-17 13:22:22 -07:00
• 950ed8fde9 regex scanner groups findall tuple bug fixed regex_scanner sohyeonhwang 2019-12-12 07:47:07 -06:00
• 097c60a7bc handling empty text sohyeonhwang 2019-12-03 16:44:53 -06:00
• cdfa77d66d remove commented code master Nathan TeBlunthuis 2019-11-11 11:28:48 -08:00
• 02b3250a36 refactor regex matching in a tidier object oriented style Nathan TeBlunthuis 2019-11-09 13:07:46 -08:00
• 414cc5ff2d validate tests and add asserts and baselines for regex tests. Nathan TeBlunthuis 2019-11-09 12:19:55 -08:00
• 4ccde84529 added regex scanner v2's dump unit test file regextest.xml.bz2 sohyeonhwang 2019-11-07 14:06:15 -06:00
• f147e1d899 merging pull containing revert-radius with 2nd version of regex scanner w/ unit tests sohyeonhwang 2019-11-07 13:28:17 -06:00
• c84844cfb5 add unit tests for configuring revert_radius groceryheist 2019-10-07 15:02:30 -07:00
• c4416d0f1b make revert radius configurable groceryheist 2019-10-07 13:57:49 -07:00
• 7b856bec86 Merge branch 'master' into regex_scanner groceryheist 2019-10-05 18:17:03 -07:00
• 324ccc8e26 update baseline outputs groceryheist 2019-10-05 16:36:07 -07:00
• 17529cdd48 bugfix, remove old legacy persistence flag groceryheist 2019-10-05 16:13:11 -07:00
• 7bf4559ceb changes for regex scanner addition sohyeonhwang 2019-10-05 15:36:58 -05:00
• fb052ffa33 edont compute persistence by default groceryheist 2019-09-22 15:54:17 -07:00
• e871023ff5 elaborate docstring for persistence groceryheist 2019-09-22 15:11:59 -07:00
• 2d5008113b add flag for excluding whitespace and punctuation tests groceryheist 2018-12-12 16:38:47 -08:00
• 19eda6dd0e use only a part of the sailormoon wiki groceryheist 2018-12-12 16:12:57 -08:00
• 4089ebae92 create state where all tests pass groceryheist 2018-12-12 16:08:00 -08:00
• 1b81b9542d make sailormoon smaller groceryheist 2018-12-12 14:57:03 -08:00
• 9c5a1b18f0 add test files groceryheist 2018-12-12 14:56:48 -08:00
• 26ea272114 wikiq mostly functional, but reverters take all the credit for the content they restore. groceryheist 2018-12-11 19:36:49 -08:00
• 0c2d72b881 checking in work to deepen migration to new mediawikiutils groceryheist 2018-12-11 00:31:50 -08:00
• 7d62ff9fb7 improve help for namespace-include groceryheist 2018-09-03 11:30:12 -07:00
• f7f5bf8fd4 sub assertEquals assertEqual advanced_persistence groceryheist 2018-09-03 11:21:49 -07:00
• f784c77f60 add namespace filter parameter Nate E TeBlunthuis 2018-08-23 18:25:08 -07:00
• df18d6e280 Merge branch 'user_level_wikiq' of code.communitydata.cc:mediawiki_dump_tools into user_level_wikiq user_level_wikiq groceryheist 2018-08-31 16:03:07 -07:00
• 3af71f03e0 Merge branch 'user_level_wikiq' of code.communitydata.cc:mediawiki_dump_tools into user_level_wikiq groceryheist 2018-08-31 16:02:05 -07:00
• 1d5a9b53b8 Merge branch 'user_level_wikiq' of code.communitydata.cc:mediawiki_dump_tools into user_level_wikiq groceryheist 2018-08-31 16:02:05 -07:00
• cc551eef6e Merge branch 'user_level_wikiq' of code.communitydata.cc:mediawiki_dump_tools into user_level_wikiq groceryheist 2018-08-31 16:01:07 -07:00
• 4c77c0f12e Merge branch 'user_level_wikiq' of code.communitydata.cc:mediawiki_dump_tools into user_level_wikiq groceryheist 2018-08-31 16:01:07 -07:00
• 915c864ee5 add more variables and support for persistence groceryheist 2018-08-31 15:57:48 -07:00
• 3d12865c4e add more variables and support for persistence groceryheist 2018-08-31 15:57:48 -07:00
• bc1f5428f0 add spark program for running group by users groceryheist 2018-08-31 20:40:22 +00:00
• 317bafb50d Merge branch 'advanced_persistence' of code.communitydata.cc:mediawiki_dump_tools into advanced_persistence groceryheist 2018-08-23 18:52:54 -07:00
• 7cd0bf3b9e Add parameter for selecting specific namespaces. groceryheist 2018-08-23 18:27:09 -07:00
• d93769c21f Merge branch 'advanced_persistence' of code.communitydata.cc:mediawiki_dump_tools into advanced_persistence groceryheist 2018-08-23 18:27:09 -07:00
• afd40c1a45 Merge branch 'advanced_persistence' of code.communitydata.cc:/mediawiki_dump_tools into advanced_persistence Nate E TeBlunthuis 2018-08-23 18:23:36 -07:00
• e4222c45dd add namespace filter parameter Nate E TeBlunthuis 2018-08-23 18:25:08 -07:00
• 829ffcffae Merge branch 'advanced_persistence' of code.communitydata.cc:/mediawiki_dump_tools into advanced_persistence Nate E TeBlunthuis 2018-08-23 18:23:36 -07:00
• 776b73519a add namespace filter parameter Nate E TeBlunthuis 2018-08-23 18:02:56 -07:00
• 5b6aaad862 add namespace filter parameter Nate E TeBlunthuis 2018-08-23 18:02:56 -07:00
• f468d1a5b6 add support for persistence with segment matching groceryheist 2018-08-20 16:08:16 -07:00
• ff689c71dd Merge branch 'user_level_wikiq' of code.communitydata.cc:mediawiki_dump_tools into user_level_wikiq groceryheist 2018-08-14 14:44:37 -07:00
• 39b4e5698f Use dask to parallelize and scale user level datasets groceryheist 2018-08-14 14:37:03 -07:00
• daf1851cbb Use dask to parallelize and scale user level datasets groceryheist 2018-08-14 14:37:03 -07:00
• 418fa020e5 Merge branch 'user_level_wikiq' of code.communitydata.cc:mediawiki_dump_tools into user_level_wikiq groceryheist 2018-08-12 21:34:12 -07:00
• 311810a36c refactor wikiq to seperate script from classes and functions. Code reuse in testing. groceryheist 2018-08-12 21:33:19 -07:00
• 118b8b1722 move tests to test folder groceryheist 2018-08-12 18:05:59 -07:00
• f69e8b44a6 move tests to test folder groceryheist 2018-08-12 18:05:59 -07:00
• bf396ad366 Prefix page titles with namespace names. mediawiki-utils-migration groceryheist 2018-07-09 22:11:17 -07:00
• d1f5e7b44c undoing my changes to master for now. see branch mediawiki-utils-migration legacy groceryheist 2018-07-05 01:40:17 -07:00
• dba793c6ac migrate to mwxml. This completes the migration away from python-mediawiki-utilities. Except for preserving legacy persistence behavior, we can safely use the nice updates from the mediawiki-utils project. groceryheist 2018-07-05 01:16:00 -07:00
• d77b0a4965 migrate to mwpersistence. this fixes many issues. We preserve legacy persistence behavior using the --persistence-legacy. groceryheist 2018-07-04 19:06:07 -07:00
• 7db6288923 migrate reverts to python-mwreverts groceryheist 2018-07-04 15:29:48 -07:00
• a883cb536b add note to readme about dependency on compression software groceryheist 2018-07-04 15:20:52 -07:00
• e925ac9da1 add tests for wikipedia, malformed xml, bzip2, correct bz2 bug in wikiq. groceryheist 2018-07-04 15:08:30 -07:00
• d2746879d0 create baseline tests for xml dump processing groceryheist 2018-07-03 23:43:47 -07:00
• ba886ecf4c a number of small updates and fixes Benjamin Mako Hill 2018-05-17 14:37:20 -07:00
• 3f9da40747 support 7z archives with multiple files. add urlencode paraeter groceryheist 2017-12-07 15:10:56 -08:00
• 5d7dceb9e4 fix code to work with bzip files Benjamin Mako Hill 2017-02-06 18:25:17 -08:00
• 7d8ec932dd added list of compressed dump files to .gitignore Benjamin Mako Hill 2015-07-23 12:16:31 -07:00
• d934700ee9 added support to parse namespaces from title Benjamin Mako Hill 2015-07-23 12:12:20 -07:00
• 108c8442b2 added README file to document the submodule Benjamin Mako Hill 2015-07-22 19:55:08 -07:00
• eeb0742cc6 created new repository for wikiq with Mediawiki-Utilities as a submodule Benjamin Mako Hill 2015-07-22 19:44:52 -07:00