diff --git a/datasets/README.md b/datasets/README.md index 7868cb9..ae305ae 100644 --- a/datasets/README.md +++ b/datasets/README.md @@ -60,9 +60,6 @@ runs the Part 2 Spark sort. ### Add new months — `add_months.sh YYYY-MM [YYYY-MM ...]` -> **NOTE: written but not yet tested. Remove this notice after a -> successful end-to-end run.** - Use this for routine incremental updates. Runs Part 1 on only the specified months, then appends the sorted output as a new layer of partition files alongside the existing ones. No existing data is @@ -130,9 +127,6 @@ from a login node — it takes the number of nodes as its first argument. ### Merge layers — `merge_layers.sh` -> **NOTE: written but not yet tested. Remove this notice after a -> successful end-to-end run.** - Use this to collapse accumulated layers from incremental adds into a single clean layer. Reads the existing final datasets, re-sorts everything, writes to `.merging` temp paths, then atomically replaces the