remove parquet output support

Browse Source

Nate has given up on getting wikiq to output parquet directly because it
makes too many unpredictable large memory allocations. The workflow now
outputs JSONL in a single pass and then uses spark (wikiq_spark) to index
it as parquet in a second pass.

Remove the parquet output path along with the machinery that existed only
to support it: checkpoint files, resume temp-file merging, namespace
partitioning, and file rotation (--partition-namespaces,
--max-revisions-per-file). Resume support remains for JSONL output, where
the resume point is derived from the last complete line of the output
file. Also remove the parquet tests and baseline files.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 72a8851373)

...

This commit is contained in:

Benjamin Mako Hill

2026-08-06 15:31:48 -07:00

parent caf604e43d

commit 015d4f9164

8 changed files with 39 additions and 875 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

BIN

test/baseline_output/collapse-user_sailormoon.parquet

Binary file not shown.

BIN

test/baseline_output/diff_sailormoon.parquet

Binary file not shown.

BIN

test/baseline_output/text_sailormoon.parquet

Binary file not shown.