collective/cdsc_reddit
20
0
Fork 0
Code Activity

datasets/add_months.sh: set PYSPARK_PYTHON so spark workers use the venv

Browse Source 
spark-submit defaults to the container Python, which doesn't have the
project's dependencies. Export PYSPARK_PYTHON=$PYTHON before starting
the cluster so workers use the same interpreter as Part 1.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Benjamin Mako Hill
2026-05-25 22:16:50 -07:00
parent 2390d2d10c
commit 7c9db519ac
1 changed files with 1 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
datasets/add_months.sh
View File

@@ -108,6 +108,7 @@ parallel --joblog add_months_joblog.txt --results add_months_logs \
# --- Part 2: sort new months into staging (Spark, single fat node) ---------- # --- Part 2: sort new months into staging (Spark, single fat node) ----------
source "$SPARK_CONF_DIR/spark-env.sh" source "$SPARK_CONF_DIR/spark-env.sh"
export PYSPARK_PYTHON="$PYTHON"
start_spark_cluster.sh start_spark_cluster.sh
spark-submit --master "spark://$(hostname):$SPARK_MASTER_PORT" \ spark-submit --master "spark://$(hostname):$SPARK_MASTER_PORT" \