collective/cdsc_reddit
20
0
Fork 0
Code Activity

datasets: explicit partition counts and cache df between sorts

Browse Source 
sort_and_write() now takes num_partitions (default from config or 200)
and passes it explicitly to repartition() instead of relying on
spark.sql.shuffle.partitions. comments_part2 defaults to 800 partitions
(~1GB/file matching the existing dataset density) and submissions_part2
to 300. Also cache the preprocessed DataFrame so the input is not
re-read and reshuffled twice when writing the by_subreddit and
by_author outputs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Benjamin Mako Hill
2026-05-29 14:48:05 -07:00
parent 9bc871285b
commit 172e66865c
3 changed files with 23 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
datasets/comments_part2.py
View File

@@ -15,7 +15,9 @@ from dumps_helper import COMMENTS, sort_and_write
if __name__ == "__main__":
fire.Fire(lambda indir=None, out_by_subreddit=None, out_by_author=None:
fire.Fire(lambda indir=None, out_by_subreddit=None, out_by_author=None,
num_partitions=800:
sort_and_write(COMMENTS, indir=indir,
out_by_subreddit=out_by_subreddit,
out_by_author=out_by_author))
out_by_author=out_by_author,
num_partitions=num_partitions))