Benjamin Mako Hill 172e66865c datasets: explicit partition counts and cache df between sorts ...

sort_and_write() now takes num_partitions (default from config or 200)
and passes it explicitly to repartition() instead of relying on
spark.sql.shuffle.partitions. comments_part2 defaults to 800 partitions
(~1GB/file matching the existing dataset density) and submissions_part2
to 300. Also cache the preprocessed DataFrame so the input is not
re-read and reshuffled twice when writing the by_subreddit and
by_author outputs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-05-29 14:48:05 -07:00

963 B

Executable File

Raw Blame History

View Raw