Will Beason
9ee5ecfc91
Separate revision iteration and field collation logic
...
This way we're not adding temporary fields to objects that don't
normally have these fields.
Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-30 14:09:16 -05:00
Will Beason
f9383440a0
Fix tests
...
Surprisingly replacing list<str> with str doesn't break anything,
even baselines.
Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-30 13:56:31 -05:00
Will Beason
032fec3198
Remove unnecessary urlencode tests
...
Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-30 13:20:10 -05:00
Will Beason
0d56267ae0
Get parquet libraries writing files
...
Tests broken due to url encoding, which can likely now be removed.
Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-30 13:06:26 -05:00
Nathan TeBlunthuis
260e2b177c
fix order of fields.
2025-05-29 18:32:16 -07:00
Nathan TeBlunthuis
a13d7f1deb
typo fix.
2025-05-29 18:25:08 -07:00
Nathan TeBlunthuis
ffbd180001
make editorid null not '' in parquet.
2025-05-29 18:24:33 -07:00
Nathan TeBlunthuis
606a399450
handle empty comments which are 'False' somehow.
2025-05-29 18:14:58 -07:00
Nathan TeBlunthuis
a9f76a0f62
change order of fields.
2025-05-29 18:10:59 -07:00
Nathan TeBlunthuis
f39ceefa4a
Merge branch 'parquet_support' of gitea:collective/mediawiki_dump_tools into parquet_support
2025-05-29 18:05:28 -07:00
Nathan TeBlunthuis
13ee160708
bugfix.
2025-05-29 18:04:41 -07:00
Nathan TeBlunthuis
bd22d26291
update deps and add edit_summary to wikiq output.
2025-05-29 18:02:14 -07:00
Will Beason
4dde25c508
Refactor revision logic to make more straightforward
...
Use groupby so we don't have to deal with edge cases and compare
revisions directly.
Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-29 15:46:45 -05:00
Will Beason
aec6e5fafa
Refactor collapse user logic
...
Use simple loop for when we aren't collapsing users.
Add test which covers case when users are deleted.
Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-29 15:20:34 -05:00
Will Beason
c0e629a313
Add ability to disable revert detection
...
Also add test to ensure functionality works.
Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-29 11:59:10 -05:00
Will Beason
9009bb6fa4
Merge branch 'parquet_support' into test-parquet
2025-05-29 10:21:30 -05:00
Will Beason
ab280dd765
Remove requirements.txt and add uv.lock to ignored files.
...
We can choose to check in uv.lock later if we want.
Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-29 10:05:49 -05:00
Nathan TeBlunthuis
22d14dc5f2
Remove dependency on pytest.
2025-05-28 21:54:31 -07:00
Nathan TeBlunthuis
5a10f59dc4
Merge branch 'parquet_support' of gitea:collective/mediawiki_dump_tools into parquet_support
2025-05-28 23:52:59 -05:00
Nathan TeBlunthuis
b8cdc82fc2
add ipython for dev
2025-05-28 23:52:37 -05:00
Nathan TeBlunthuis
2a2b611d79
Fix issue with .7z archives
...
Before, only fandom wikis dumps were compressed with .7z.
These archives can have several .xml files in the .7z; not just one.
So we need to have a flag for the fandom-2020 dumps.
This fixes the bug so .7z archives work in either case.
2025-05-28 21:49:11 -07:00
Nathan TeBlunthuis
39fec0820d
use my version of mwxml since it fixes a bug.
2025-05-28 21:13:18 -07:00
Nathan TeBlunthuis
383ee03250
Merge branch 'parquet_support' of gitea:collective/mediawiki_dump_tools into parquet_support
2025-05-28 21:09:13 -07:00
Nathan TeBlunthuis
15e9234903
adding pyproject.toml
2025-05-28 20:59:55 -07:00
Nathan TeBlunthuis
8c7d46472f
Merge branch 'parquet_support' of code:mediawiki_dump_tools into parquet_support
2025-05-28 20:54:52 -07:00
Nathan TeBlunthuis
3c7fb088d6
fix schema bugs.
2025-05-28 20:54:42 -07:00
Will Beason
ee01ce3e61
Get Parquet test working
...
This requires some data smoothing to get read_table and read_parquet
DataFrames to look close enough, but the test now passes and validates
that the data match.
Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-28 16:48:58 -05:00
Will Beason
52757a8239
Add noargs test for ikwiki
...
This way we can ensure that the parquet code outputs equivalent output.
Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-28 15:04:10 -05:00
Will Beason
d413443740
Add numpy to environment
...
Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-28 13:20:28 -05:00
Will Beason
3f94144b1b
Begin adding test for parquet export
...
Changed logic for handling anonymous edits so that wikiq handles
the type for editor ids consistently. Parquet can mix int64 and
None, but not int64 and strings - previously the code used the empty
string to denote anonymous editors.
Tests failing. Don't merge yet.
Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-28 13:17:30 -05:00
Will Beason
df0ad1de63
Finish test standardization
...
Test logic is executed within the WikiqTestCase, while WikiqTester
handles creating and managing the variables tests need.
Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-28 10:11:58 -05:00
Will Beason
f3e6cc9392
Begin refactor of tests to make new tests easier to write
...
Handle file naming logic centrally rather than requiring a dedicated
class per input file.
Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-28 09:11:36 -05:00
Will Beason
c8b14c3303
Refactor test temporary file logic and wikiq call pattern
...
Test file refreshing and path computation is now handled by a helper.
The wikiq command is now constructed and handled by a single method
rather than in several ad-hoc ways.
The last places relying on the working directory are now removed.
Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-27 16:24:07 -05:00
Will Beason
4d3900b541
Standardize calling for wikiq in tests
...
This way failures show the output of stderr/etc.
Also create path constant strings for use in tests to avoid repetition
and make changes easier.
Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-27 14:27:49 -05:00
Will Beason
ebc57864f2
Make tests runnable from anywhere
...
Tests no longer implicitly require that the caller be in
a specific working directory.
Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-27 13:40:57 -05:00
Will Beason
3d0bf89938
Move main logic to main()
...
This avoids:
1) the main function running when sourcing the file
2) Creating many globally-scoped variables in the main logic
Also begin refactor of test output file logic
Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-27 11:10:42 -05:00
Will Beason
6d133575c7
Remove resource leaks from tests
...
Close subprocesses within tests to fix resource leak warning.
Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-26 15:08:47 -05:00
Will Beason
09a84e7d11
Reformat Wikiq_Unit_Test.py
...
Separate out reformatting from editing.
Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-26 15:07:39 -05:00
Will Beason
9c5bf577e6
Remove unused dependencies and fix spacing
...
The "mw" and "numpy" dependencies were unneeded.
Spaces and tabs were inconsistently used.
They are now used consistently, changes via auto-formatter.
Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-26 14:15:01 -05:00
Will Beason
4804ecc4b3
Add additional test dependencies
...
These are now noted in requirements.txt
Also make dependency on 7zip and ffmpeg explicit in README
Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-26 12:29:49 -05:00
Will Beason
7a4c41159c
Exclude JetBrains config folder in .gitignore
...
Signed-off-by: Will Beason <willbeason@gmail.com>
2025-05-26 10:48:17 -05:00
1aea601a30
[Bugfix] Call the correct matchmake function.
2021-11-16 16:53:21 -08:00
c437b357db
rename matchmake functions
2021-11-11 19:09:41 -08:00
bb83d62b74
Add some descriptive comments.
2021-10-19 16:55:24 -07:00
c285402683
add todos to readme
2021-10-18 14:14:11 -07:00
b1bea09ad6
fix bugs and unit tests
2021-10-18 13:33:05 -07:00
9a0c157ebb
bugfix
2021-10-18 10:15:03 -07:00
ae870fed0b
parquet path is code-complete
2021-10-17 21:46:31 -07:00
26f6d8f984
remove dependency on pandas.
2021-10-17 20:24:33 -07:00
ae9a241747
use dataclasses and pyarrow for types.
2021-10-17 20:21:22 -07:00
