collective/cdsc_reddit
20
0
Fork 0
Code Activity

datasets/README: remove untested notices after successful end-to-end run

Browse Source 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Benjamin Mako Hill
2026-07-21 10:05:18 -07:00
parent 172e66865c
commit 1525a48561
1 changed files with 0 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
datasets/README.md
View File

@@ -60,9 +60,6 @@ runs the Part 2 Spark sort.
### Add new months — `add_months.sh YYYY-MM [YYYY-MM ...]` ### Add new months — `add_months.sh YYYY-MM [YYYY-MM ...]`
> **NOTE: written but not yet tested. Remove this notice after a
> successful end-to-end run.**
Use this for routine incremental updates. Runs Part 1 on only the Use this for routine incremental updates. Runs Part 1 on only the
specified months, then appends the sorted output as a new layer of specified months, then appends the sorted output as a new layer of
partition files alongside the existing ones. No existing data is partition files alongside the existing ones. No existing data is
@@ -130,9 +127,6 @@ from a login node — it takes the number of nodes as its first argument.
### Merge layers — `merge_layers.sh` ### Merge layers — `merge_layers.sh`
> **NOTE: written but not yet tested. Remove this notice after a
> successful end-to-end run.**
Use this to collapse accumulated layers from incremental adds into a Use this to collapse accumulated layers from incremental adds into a
single clean layer. Reads the existing final datasets, re-sorts single clean layer. Reads the existing final datasets, re-sorts
everything, writes to `.merging` temp paths, then atomically replaces the everything, writes to `.merging` temp paths, then atomically replaces the