setting up the environment by loading in conda environment at Sat Oct 11 07:52:03 CDT 2025
running the batched olmo categorization job at Sat Oct 11 07:52:03 CDT 2025
[nltk_data] Downloading package punkt_tab to
[nltk_data]     /home/nws8519/nltk_data...
[nltk_data]   Package punkt_tab is already up-to-date!
cuda
NVIDIA A100-SXM4-80GB
_CudaDeviceProperties(name='NVIDIA A100-SXM4-80GB', major=8, minor=0, total_memory=81153MB, multi_processor_count=108, uuid=ee0bd2a7-af54-5f2e-c2d3-fcd3f57270c9, L2_cache_size=40MB)

Loading checkpoint shards:   0%|          | 0/12 [00:00<?, ?it/s]
Loading checkpoint shards:   8%|▊         | 1/12 [00:00<00:04,  2.72it/s]
Loading checkpoint shards:  17%|█▋        | 2/12 [00:00<00:04,  2.27it/s]
Loading checkpoint shards:  25%|██▌       | 3/12 [00:01<00:04,  2.12it/s]
Loading checkpoint shards:  33%|███▎      | 4/12 [00:01<00:03,  2.12it/s]
Loading checkpoint shards:  42%|████▏     | 5/12 [00:02<00:03,  1.96it/s]
Loading checkpoint shards:  50%|█████     | 6/12 [00:02<00:03,  1.98it/s]
Loading checkpoint shards:  58%|█████▊    | 7/12 [00:03<00:02,  1.87it/s]
Loading checkpoint shards:  67%|██████▋   | 8/12 [00:03<00:02,  1.94it/s]
Loading checkpoint shards:  75%|███████▌  | 9/12 [00:04<00:01,  1.91it/s]
Loading checkpoint shards:  83%|████████▎ | 10/12 [00:05<00:01,  1.87it/s]
Loading checkpoint shards:  92%|█████████▏| 11/12 [00:05<00:00,  2.05it/s]
Loading checkpoint shards: 100%|██████████| 12/12 [00:05<00:00,  2.18it/s]
Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.
This is a friendly reminder - the current text generation call will exceed the model's predefined maximum length (4096). Depending on the model, you may observe exceptions, performance degradation, or nothing at all.
Traceback (most recent call last):
  File "/home/nws8519/git/mw-lifecycle-analysis/p2/quest/python_scripts/090425_batched_olmo_cat.py", line 66, in <module>
    for row in reader:
_csv.Error: field larger than field limit (131072)
unsupervised batched olmo categorization pau at Sun Oct 12 14:11:43 CDT 2025