Commit Graph

26 Commits

Author SHA1 Message Date
Will Beason
3f94144b1b Begin adding test for parquet export
Changed logic for handling anonymous edits so that wikiq handles
the type for editor ids consistently. Parquet can mix int64 and
None, but not int64 and strings - previously the code used the empty
string to denote anonymous editors.

Tests failing. Don't merge yet.

Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-28 13:17:30 -05:00
Will Beason
df0ad1de63 Finish test standardization
Test logic is executed within the WikiqTestCase, while WikiqTester
handles creating and managing the variables tests need.

Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-28 10:11:58 -05:00
Will Beason
f3e6cc9392 Begin refactor of tests to make new tests easier to write
Handle file naming logic centrally rather than requiring a dedicated
class per input file.

Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-28 09:11:36 -05:00
Will Beason
c8b14c3303 Refactor test temporary file logic and wikiq call pattern
Test file refreshing and path computation is now handled by a helper.

The wikiq command is now constructed and handled by a single method
rather than in several ad-hoc ways.

The last places relying on the working directory are now removed.

Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-27 16:24:07 -05:00
Will Beason
4d3900b541 Standardize calling for wikiq in tests
This way failures show the output of stderr/etc.

Also create path constant strings for use in tests to avoid repetition
and make changes easier.

Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-27 14:27:49 -05:00
Will Beason
ebc57864f2 Make tests runnable from anywhere
Tests no longer implicitly require that the caller be in
a specific working directory.

Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-27 13:40:57 -05:00
Will Beason
3d0bf89938 Move main logic to main()
This avoids:
1) the main function running when sourcing the file
2) Creating many globally-scoped variables in the main logic

Also begin refactor of test output file logic

Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-27 11:10:42 -05:00
Will Beason
6d133575c7 Remove resource leaks from tests
Close subprocesses within tests to fix resource leak warning.

Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-26 15:08:47 -05:00
Will Beason
09a84e7d11 Reformat Wikiq_Unit_Test.py
Separate out reformatting from editing.

Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-26 15:07:39 -05:00
Will Beason
4804ecc4b3 Add additional test dependencies
These are now noted in requirements.txt

Also make dependency on 7zip and ffmpeg explicit in README

Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-26 12:29:49 -05:00
b1bea09ad6 fix bugs and unit tests 2021-10-18 13:33:05 -07:00
414cc5ff2d validate tests and add asserts and baselines for regex tests. 2019-11-09 12:19:55 -08:00
sohyeonhwang
4ccde84529 added regex scanner v2's dump unit test file regextest.xml.bz2 2019-11-07 14:06:15 -06:00
sohyeonhwang
f147e1d899 merging pull containing revert-radius with 2nd version of regex scanner w/ unit tests 2019-11-07 13:28:17 -06:00
c84844cfb5 add unit tests for configuring revert_radius 2019-10-07 15:02:30 -07:00
7b856bec86 Merge branch 'master' into regex_scanner 2019-10-05 18:17:03 -07:00
324ccc8e26 update baseline outputs 2019-10-05 16:36:07 -07:00
sohyeonhwang
7bf4559ceb changes for regex scanner addition 2019-10-05 15:36:58 -05:00
f7f5bf8fd4 sub assertEquals assertEqual 2018-09-03 11:21:49 -07:00
7cd0bf3b9e Add parameter for selecting specific namespaces. 2018-08-23 18:49:32 -07:00
f468d1a5b6 add support for persistence with segment matching 2018-08-20 16:08:16 -07:00
bf396ad366 Prefix page titles with namespace names. 2018-07-09 22:11:17 -07:00
dba793c6ac migrate to mwxml. This completes the migration away from python-mediawiki-utilities. Except for preserving legacy persistence behavior, we can safely use the nice updates from the mediawiki-utils project. 2018-07-05 01:16:00 -07:00
d77b0a4965 migrate to mwpersistence. this fixes many issues. We preserve legacy persistence behavior using the --persistence-legacy. 2018-07-04 19:06:07 -07:00
e925ac9da1 add tests for wikipedia, malformed xml, bzip2, correct bz2 bug in wikiq. 2018-07-04 15:08:30 -07:00
d2746879d0 create baseline tests for xml dump processing 2018-07-03 23:43:47 -07:00