Benjamin Mako Hill 9bc871285b datasets/add_months.sh: auto-detect cores/memory for spark-submit ...

Compute executor cores and memory from the values spark-env.sh detects
at startup (SPARK_WORKER_CORES, SPARK_WORKER_MEMORY) and pass them as
--conf flags to spark-submit. This overrides the static defaults in
spark-defaults.conf so the job scales automatically to whatever machine
it runs on. Uses a bash array to avoid word-splitting issues.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-05-26 10:07:58 -07:00

6.8 KiB

Executable File

Raw Blame History

View Raw