Benjamin Mako Hill
2d1d760142
Add add_months.sh and merge_layers.sh implementing a layered append strategy for incremental dataset updates. Each incremental run appends new sorted partition files alongside existing ones rather than re-sorting the full corpus, which is prohibitively slow at this dataset scale. - dumps_helper.py: sort_and_write gains indir/mode params; new merge_layers function collapses accumulated layers via atomic rename - comments_part2.py, submissions_part2.py: expose --indir/--mode via Fire - add_months.sh: new layered append script (not yet tested) - merge_layers.sh: new layer collapse script (not yet tested) - comments_merge.py, submissions_merge.py: Spark entry points for merge - add_new_month.sh: deleted (full re-sort each add is redundant with build_from_scratch at corpus scale) - README.md: document three workflows; flag untested sections Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
19 lines
681 B
Python
Executable File
19 lines
681 B
Python
Executable File
#!/usr/bin/env python3
|
|
"""Part 2 for comments: Spark sort + repartition into the final datasets.
|
|
|
|
Must be launched from a login node via the Hyak-provided wrapper:
|
|
start_spark_and_run.sh 1 comments_part2.py
|
|
start_spark_and_run.sh 1 comments_part2.py --indir=/path/to/parquets --mode=append
|
|
|
|
--indir defaults to the temp comments dir in dumps_helper.py.
|
|
--mode defaults to 'overwrite'; use 'append' to add a new layer without
|
|
touching existing partition files (see add_months.sh).
|
|
"""
|
|
|
|
import fire
|
|
from dumps_helper import COMMENTS, sort_and_write
|
|
|
|
|
|
if __name__ == "__main__":
|
|
fire.Fire(lambda indir=None, mode='overwrite': sort_and_write(COMMENTS, indir=indir, mode=mode))
|