collective/cdsc_reddit
18
0
Fork 0
Code Activity

datasets/add_months.sh: fail on leftover files, add --clean to wipe them

Browse Source 
Without --clean, the script now exits with a clear error if temp or
staging directories from a previous run exist. Pass --clean to remove
them automatically before starting. README example updated to include
the flag.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Benjamin Mako Hill
2026-05-25 19:10:16 -07:00
parent 6c6e05c360
commit 0ea57b2377
2 changed files with 35 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
datasets/README.md
View File

@@ -115,7 +115,7 @@ srun -p cpu-g2 -A comdata --nodes=1 --time=72:00:00 -c 112 --mem=400G \
PYTHON=/gscratch/comdata/users/makohill/cdsc_reddit/venv/bin/python3 \
COMMENTS_DUMPDIR=/path/to/new/comments \
SUBMISSIONS_DUMPDIR=/path/to/new/submissions \
./datasets/add_months.sh 2025-01 2025-02 ... YYYY-MM
./datasets/add_months.sh --clean 2025-01 2025-02 ... YYYY-MM
" 2>&1 | tee /gscratch/comdata/users/makohill/add_months_run.log
```