mgaughan/mw-lifecycle-analysis
1
0
Fork 0
Code

cleaning working directory and re-running PCA with final neurobiber vectors

Browse Source
This commit is contained in:
mgaughan 2025-09-25 09:48:23 -05:00
parent 9d1359af36
commit e29d4bf59c
14 changed files with 3501 additions and 100647 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

96921
p2/quest/090425_subcomment_PCA_df.csv
View File

File diff suppressed because one or more lines are too long

6464
p2/quest/090425_description_PCA_df.csv → p2/quest/092325_description_PCA_df.csv
View File

File diff suppressed because it is too large Load Diff

BIN
p2/quest/092525_description_pca.pkl Normal file
View File

Binary file not shown.

264
p2/quest/092525_neurobiber-pca.log Normal file
View File

@ -0,0 +1,264 @@
starting the job at: Thu Sep 25 09:36:43 CDT 2025
setting up the environment
running the neurobiber labeling script
Variance of each PCA component: [44.08472997 25.31736287 20.0163717 11.80556907 8.85200058 8.36660391
7.01387796 5.63183426 5.24412874 4.94069135 4.84193359 3.82172419
3.45601167 2.49140408 2.30451175 2.05674872 1.83560413 1.81691694]
PC1:
BIN_CAP: 0.573
BIN_NNP: 0.571
BIN_DET: -0.294
BIN_ART: -0.230
BIN_PIN: -0.226
BIN_PREP: -0.226
BIN_RB: -0.127
BIN_INF: -0.109
BIN_PRP: -0.106
BIN_SBJP: -0.106
PC2:
BIN_PIN: 0.493
BIN_PREP: 0.493
BIN_NN: 0.473
BIN_CAP: 0.340
BIN_NNP: 0.304
BIN_DET: 0.149
BIN_ART: 0.114
BIN_NOMZ: -0.107
BIN_INF: 0.093
BIN_CONJ: 0.073
PC3:
BIN_NN: 0.803
BIN_PREP: -0.242
BIN_PIN: -0.242
BIN_NNP: -0.232
BIN_PRP: -0.195
BIN_SBJP: -0.195
BIN_RB: -0.173
BIN_INF: -0.131
BIN_FPP1: -0.091
BIN_VPRT: -0.089
PC4:
BIN_DET: 0.588
BIN_ART: 0.529
BIN_PREP: -0.287
BIN_PIN: -0.287
BIN_CAP: 0.245
BIN_INDA: 0.186
BIN_VPRT: 0.173
BIN_JJ: -0.142
BIN_NNP: 0.122
BIN_NOMZ: -0.121
PC5:
BIN_CAP: 0.443
BIN_RB: 0.405
BIN_NNP: -0.371
BIN_PRP: 0.300
BIN_SBJP: 0.300
BIN_VPRT: 0.213
BIN_ART: -0.211
BIN_DET: -0.190
BIN_NN: 0.183
BIN_FPP1: 0.154
PC6:
BIN_JJ: 0.539
BIN_CAP: 0.393
BIN_NOMZ: 0.391
BIN_NNP: -0.353
BIN_NN: -0.210
BIN_X: -0.170
BIN_QUOT: -0.166
BIN_RB: -0.163
BIN_ART: 0.132
BIN_NUM: -0.131
PC7:
BIN_JJ: 0.573
BIN_NNP: 0.397
BIN_VPRT: 0.377
BIN_CAP: -0.305
BIN_RB: 0.259
BIN_QUOT: -0.233
BIN_X: -0.185
BIN_INF: -0.168
BIN_AUXB: 0.161
BIN_NN: 0.094
PC8:
BIN_INF: 0.732
BIN_QUOT: -0.313
BIN_VPRT: -0.246
BIN_RB: 0.203
BIN_TO: 0.192
BIN_NUM: -0.164
BIN_NOMZ: 0.158
BIN_NNP: 0.141
BIN_PREP: -0.125
BIN_PIN: -0.125
PC9:
BIN_QUOT: 0.698
BIN_JJ: 0.410
BIN_INF: 0.305
BIN_CONT: 0.292
BIN_NOMZ: -0.266
BIN_PRP: -0.125
BIN_SBJP: -0.125
BIN_X: 0.107
BIN_RB: 0.076
BIN_NUM: -0.071
PC10:
BIN_RB: 0.513
BIN_PRP: -0.422
BIN_SBJP: -0.422
BIN_FPP1: -0.203
BIN_NNP: -0.192
BIN_X: 0.184
BIN_INF: -0.176
BIN_NUM: 0.154
BIN_NOMZ: -0.151
BIN_JJ: -0.148
PC11:
BIN_X: 0.628
BIN_NOMZ: -0.458
BIN_QUOT: -0.367
BIN_JJ: 0.319
BIN_NUM: 0.173
BIN_CONT: -0.172
BIN_RB: -0.137
BIN_INF: 0.114
BIN_PRP: 0.102
BIN_SBJP: 0.101
PC12:
BIN_VPRT: 0.476
BIN_X: 0.476
BIN_NUM: -0.372
BIN_AUXB: 0.360
BIN_NOMZ: 0.302
BIN_RB: -0.226
BIN_PASS: 0.170
BIN_JJ: -0.155
BIN_VBD: -0.115
BIN_BEMA: 0.104
PC13:
BIN_X: 0.426
BIN_NUM: -0.421
BIN_RB: 0.359
BIN_AUXB: -0.351
BIN_NOMZ: 0.306
BIN_VPRT: -0.254
BIN_INF: -0.219
BIN_PASS: -0.148
BIN_TO: -0.136
BIN_BEMA: -0.135
PC14:
BIN_AUXB: 0.479
BIN_VPRT: -0.447
BIN_NUM: -0.395
BIN_VBD: 0.287
BIN_CONT: -0.242
BIN_NOMZ: -0.211
BIN_PASS: 0.211
BIN_BEMA: 0.168
BIN_INF: -0.157
BIN_PGAS: 0.149
PC15:
BIN_NUM: 0.584
BIN_NOMZ: 0.435
BIN_AUXB: 0.324
BIN_VPRT: -0.210
BIN_X: 0.197
BIN_PGAS: -0.180
BIN_RB: 0.167
BIN_BEMA: 0.160
BIN_QUOT: 0.146
BIN_VBD: 0.140
PC16:
BIN_PGAS: 0.697
BIN_CONJ: -0.431
BIN_CCONJ: -0.380
BIN_SCONJ: 0.223
BIN_WH: 0.139
BIN_TO: 0.135
BIN_WZPRES: 0.130
BIN_GER: 0.090
BIN_NUM: 0.084
BIN_PASS: -0.082
PC17:
BIN_CCONJ: 0.491
BIN_PGAS: 0.452
BIN_CONJ: 0.359
BIN_CONT: -0.341
BIN_QUOT: 0.191
BIN_VPRT: 0.177
BIN_NUM: 0.176
BIN_XX0: -0.173
BIN_VBD: -0.167
BIN_SPAU: -0.146
PC18:
BIN_CCONJ: 0.666
BIN_CONJ: -0.493
BIN_CONT: 0.275
BIN_VBD: 0.170
BIN_XX0: 0.140
BIN_INDA: -0.138
BIN_ANDC: 0.138
BIN_NUM: 0.132
BIN_PRIV: -0.116
BIN_FPP1: 0.103
Top 10 PC1 values:
PC1 PC2 ... priority closed_relevance
19873 40.267200 26.528755 ... Medium False
24120 34.012764 7.436658 ... Low False
24529 33.020514 7.624464 ... Needs Triage False
25549 33.018302 7.622737 ... Medium False
24528 33.016089 7.621010 ... Needs Triage True
23238 31.348286 5.402173 ... Medium False
18729 29.627919 4.690955 ... Needs Triage True
23016 29.595518 8.870229 ... Medium False
14849 28.191116 6.625144 ... Low False
21214 28.191116 6.625144 ... Low True
[10 rows x 26 columns]
Bottom 10 PC1 values:
PC1 PC2 ... priority closed_relevance
24481 -16.862586 13.863453 ... Needs Triage True
23053 -16.174624 12.133559 ... Medium False
23838 -15.421295 13.308099 ... Low False
25791 -15.127553 14.746424 ... Medium True
7451 -14.574686 5.821303 ... Medium False
24467 -13.905417 7.936462 ... Needs Triage True
23436 -13.827143 7.507781 ... Medium False
24293 -13.667374 0.891979 ... Unbreak Now! True
11814 -13.418003 7.854756 ... Low False
968 -13.358491 0.305388 ... Needs Triage True
[10 rows x 26 columns]
Top 10 PC2 values:
PC1 PC2 ... priority closed_relevance
25606 6.196829 29.809964 ... Medium True
21956 27.542757 27.763075 ... Needs Triage True
25078 -4.462216 27.186434 ... High False
19873 40.267200 26.528755 ... Medium False
25820 -3.022591 23.093162 ... Medium True
25814 20.151634 22.681554 ... Medium True
13345 6.035595 21.910339 ... Lowest NaN
22013 6.861197 21.673434 ... Needs Triage True
23022 0.808467 21.111863 ... Medium False
21966 -7.056224 20.953599 ... Needs Triage True
[10 rows x 26 columns]
Bottom 10 PC2 values:
PC1 PC2 ... priority closed_relevance
3134 5.606805 -12.562127 ... High True
654 -0.797645 -12.364185 ... Unbreak Now! True
16289 -0.897011 -12.328128 ... Medium False
1207 4.714780 -12.127148 ... Needs Triage True
1885 15.889004 -12.071062 ... Needs Triage True
18211 6.521166 -11.920065 ... Needs Triage True
2934 0.069845 -11.739971 ... High False
25122 -1.657588 -11.388235 ... Medium True
13276 15.441209 -11.380360 ... Lowest False
2109 -2.166594 -11.371418 ... Needs Triage True
[10 rows x 26 columns]
job finished, cleaning up
job pau at: Thu Sep 25 09:37:24 CDT 2025

12
p2/quest/batched-mw-olmo-info-cat.log
View File

@ -1,12 +0,0 @@
setting up the environment by loading in conda environment at Mon Sep 22 20:07:56 CDT 2025
running the batched olmo categorization job at Mon Sep 22 20:07:57 CDT 2025
[nltk_data] Downloading package punkt_tab to
[nltk_data] /home/nws8519/nltk_data...
[nltk_data] Package punkt_tab is already up-to-date!
cuda
NVIDIA A100-SXM4-80GB
_CudaDeviceProperties(name='NVIDIA A100-SXM4-80GB', major=8, minor=0, total_memory=81153MB, multi_processor_count=108, uuid=fb10e36f-fd51-a123-6ae4-e318d24dbb3c, L2_cache_size=40MB)
Loading checkpoint shards: 0%| | 0/12 [00:00<?, ?it/s] Loading checkpoint shards: 8%|▊ | 1/12 [00:00<00:04, 2.32it/s] Loading checkpoint shards: 17%|█▋ | 2/12 [00:01<00:06, 1.64it/s] Loading checkpoint shards: 25%|██▌ | 3/12 [00:01<00:05, 1.66it/s] Loading checkpoint shards: 33%|███▎ | 4/12 [00:02<00:04, 1.64it/s] Loading checkpoint shards: 42%|████▏ | 5/12 [00:02<00:04, 1.68it/s] Loading checkpoint shards: 50%|█████ | 6/12 [00:03<00:03, 1.60it/s] Loading checkpoint shards: 58%|█████▊ | 7/12 [00:04<00:03, 1.65it/s] Loading checkpoint shards: 67%|██████▋ | 8/12 [00:04<00:02, 1.59it/s] Loading checkpoint shards: 75%|███████▌ | 9/12 [00:05<00:01, 1.56it/s] Loading checkpoint shards: 83%|████████▎ | 10/12 [00:06<00:01, 1.64it/s] Loading checkpoint shards: 92%|█████████▏| 11/12 [00:06<00:00, 1.76it/s] Loading checkpoint shards: 100%|██████████| 12/12 [00:06<00:00, 1.82it/s]
Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.
This is a friendly reminder - the current text generation call will exceed the model's predefined maximum length (4096). Depending on the model, you may observe exceptions, performance degradation, or nothing at all.
unsupervised batched olmo categorization pau at Wed Sep 24 13:25:31 CDT 2025

8
p2/quest/cleaned_biberplus-categorization.log
View File

@ -1,8 +0,0 @@
starting the job at: Tue Sep 23 16:48:14 CDT 2025
setting up the environment
running the biberplus labeling script
26024
26024
biberplus labeling pau
job finished, cleaning up
job pau at: Tue Sep 23 16:56:55 CDT 2025

BIN
p2/quest/description_AuthorWMFAffil_090425_biber_pca_final.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.4 MiB

BIN
p2/quest/description_AuthorWMFAffil_092525_biber_pca_final.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.6 MiB

264
p2/quest/neurobiber-pca.log
View File

@ -1,264 +0,0 @@
starting the job at: Tue Sep 23 16:37:06 CDT 2025
setting up the environment
running the neurobiber labeling script
Variance of each PCA component: [88.92832185 39.46471687 32.34601523 20.19544345 14.0083261 11.5837521
7.82584723 6.89064989 6.07988254 5.80726367 5.49782354 4.50587747
4.31482409 2.81997326 2.62989708 2.27205352 2.09396341 2.00076119]
PC1:
BIN_NNP: 0.760
BIN_CAP: 0.524
BIN_DET: -0.166
BIN_PREP: -0.157
BIN_PIN: -0.157
BIN_ART: -0.126
BIN_NN: -0.119
BIN_RB: -0.076
BIN_INF: -0.070
BIN_VPRT: -0.069
PC2:
BIN_PREP: 0.473
BIN_PIN: 0.473
BIN_NNP: 0.426
BIN_DET: 0.323
BIN_ART: 0.240
BIN_NOMZ: -0.233
BIN_VPRT: 0.142
BIN_RB: 0.132
BIN_SBJP: 0.119
BIN_PRP: 0.119
PC3:
BIN_CAP: 0.727
BIN_NN: 0.546
BIN_NNP: -0.363
BIN_PREP: 0.102
BIN_PIN: 0.102
BIN_DET: 0.058
BIN_ART: 0.056
BIN_SBJP: -0.048
BIN_PRP: -0.048
BIN_PRIV: 0.036
PC4:
BIN_NN: 0.659
BIN_CAP: -0.391
BIN_PRP: -0.260
BIN_SBJP: -0.260
BIN_NNP: 0.247
BIN_RB: -0.236
BIN_ART: 0.141
BIN_FPP1: -0.130
BIN_INF: -0.128
BIN_PREP: -0.127
PC5:
BIN_DET: 0.485
BIN_ART: 0.422
BIN_PIN: -0.421
BIN_PREP: -0.421
BIN_RB: 0.245
BIN_VPRT: 0.196
BIN_INDA: 0.142
BIN_NOMZ: -0.123
BIN_PRP: 0.108
BIN_SBJP: 0.108
PC6:
BIN_NOMZ: 0.368
BIN_NN: -0.345
BIN_DET: 0.344
BIN_RB: -0.339
BIN_ART: 0.326
BIN_JJ: 0.324
BIN_PRP: -0.262
BIN_SBJP: -0.262
BIN_FPP1: -0.144
BIN_INDA: 0.128
PC7:
BIN_JJ: 0.448
BIN_X: -0.439
BIN_QUOT: -0.375
BIN_NOMZ: 0.312
BIN_NN: 0.271
BIN_RB: 0.231
BIN_NUM: -0.179
BIN_VPRT: 0.179
BIN_INF: -0.169
BIN_NNP: 0.164
PC8:
BIN_RB: 0.623
BIN_PRP: -0.415
BIN_SBJP: -0.415
BIN_FPP1: -0.240
BIN_INF: 0.233
BIN_JJ: 0.150
BIN_AUXB: 0.147
BIN_NOMZ: -0.143
BIN_XX0: 0.110
BIN_SPAU: 0.103
PC9:
BIN_INF: 0.712
BIN_VPRT: -0.427
BIN_TO: 0.206
BIN_X: -0.190
BIN_AUXB: -0.179
BIN_NUM: -0.173
BIN_QUOT: -0.161
BIN_NOMZ: 0.159
BIN_CONJ: -0.122
BIN_PRIV: 0.102
PC10:
BIN_QUOT: 0.726
BIN_JJ: 0.496
BIN_CONT: 0.327
BIN_X: -0.170
BIN_NUM: -0.149
BIN_INF: 0.134
BIN_PASS: -0.080
BIN_NOMZ: -0.074
BIN_NN: 0.068
BIN_AUXB: -0.060
PC11:
BIN_X: 0.620
BIN_JJ: 0.575
BIN_NOMZ: -0.292
BIN_QUOT: -0.288
BIN_INF: 0.131
BIN_PRP: 0.125
BIN_SBJP: 0.125
BIN_CONT: -0.123
BIN_RB: -0.092
BIN_FPP1: 0.085
PC12:
BIN_VPRT: 0.529
BIN_AUXB: 0.431
BIN_RB: -0.404
BIN_INF: 0.364
BIN_TO: 0.187
BIN_ART: -0.186
BIN_PASS: 0.183
BIN_VBD: -0.158
BIN_BEMA: 0.128
BIN_DEMP: 0.110
PC13:
BIN_NUM: 0.554
BIN_X: -0.544
BIN_NOMZ: -0.509
BIN_JJ: 0.160
BIN_RB: -0.156
BIN_QUOT: -0.124
BIN_CONT: -0.109
BIN_NN: -0.103
BIN_VPRT: -0.081
BIN_NNP: -0.073
PC14:
BIN_NUM: 0.595
BIN_NOMZ: 0.366
BIN_VPRT: 0.348
BIN_AUXB: -0.332
BIN_VBD: -0.262
BIN_PASS: -0.188
BIN_CONT: 0.161
BIN_INF: 0.157
BIN_PGAS: -0.118
BIN_CONJ: -0.118
PC15:
BIN_AUXB: 0.484
BIN_NUM: 0.450
BIN_NOMZ: 0.315
BIN_VPRT: -0.307
BIN_VBD: 0.262
BIN_PASS: 0.207
BIN_BEMA: 0.194
BIN_CONJ: 0.170
BIN_PRIV: -0.162
BIN_QUOT: 0.159
PC16:
BIN_CONJ: 0.673
BIN_PGAS: -0.355
BIN_CCONJ: 0.324
BIN_SCONJ: -0.247
BIN_TO: -0.197
BIN_VBD: -0.185
BIN_WH: -0.164
BIN_FPP1: -0.128
BIN_PRIV: 0.113
BIN_DEMP: -0.096
PC17:
BIN_CCONJ: 0.471
BIN_CONT: 0.462
BIN_INDA: -0.260
BIN_XX0: 0.221
BIN_SCONJ: -0.216
BIN_CONJ: -0.210
BIN_SPAU: 0.199
BIN_DET: 0.197
BIN_FPP1: 0.196
BIN_QUOT: -0.185
PC18:
BIN_PGAS: 0.578
BIN_CCONJ: 0.564
BIN_CONT: -0.268
BIN_PRIV: -0.235
BIN_ANDC: 0.144
BIN_PASS: -0.143
BIN_QUOT: 0.138
BIN_SPAU: -0.125
BIN_VBD: -0.115
BIN_NOMZ: 0.114
Top 10 PC1 values:
PC1 PC2 ... priority closed_relevance
19873 125.128650 24.461032 ... Medium False
21956 125.128650 24.461032 ... Needs Triage True
22010 125.128650 24.461032 ... Needs Triage True
24528 125.128650 24.461032 ... Needs Triage True
24529 125.128650 24.461032 ... Needs Triage False
25549 125.128650 24.461032 ... Medium False
6329 72.728923 28.262157 ... Medium False
11288 72.728923 28.262157 ... Low False
22332 72.728923 28.262157 ... High True
22731 72.728923 28.262157 ... Medium True
[10 rows x 26 columns]
Bottom 10 PC1 values:
PC1 PC2 ... priority closed_relevance
12503 -16.333841 17.142328 ... Low NaN
3462 -15.759184 15.368325 ... High NaN
23838 -14.821270 17.471553 ... Low False
25791 -14.806017 12.439508 ... Medium True
23053 -14.399838 15.867529 ... Medium False
24180 -14.046494 12.993193 ... Low True
11814 -14.009692 13.953416 ... Low False
24699 -13.848945 15.308788 ... Needs Triage True
24214 -13.701324 11.951003 ... Low False
24467 -13.680693 11.614764 ... Needs Triage True
[10 rows x 26 columns]
Top 10 PC2 values:
PC1 PC2 PC3 ... week_index priority closed_relevance
6329 72.728923 28.262157 -52.466963 ... 10 Medium False
11288 72.728923 28.262157 -52.466963 ... 2 Low False
22332 72.728923 28.262157 -52.466963 ... 4 High True
22731 72.728923 28.262157 -52.466963 ... 10 Medium True
23016 72.728923 28.262157 -52.466963 ... 7 Medium False
23022 72.728923 28.262157 -52.466963 ... 7 Medium False
23086 72.728923 28.262157 -52.466963 ... 6 Medium False
23238 72.728923 28.262157 -52.466963 ... 4 Medium False
25606 72.728923 28.262157 -52.466963 ... -22 Medium True
25843 72.728923 28.262157 -52.466963 ... -31 Medium True
[10 rows x 26 columns]
Bottom 10 PC2 values:
PC1 PC2 PC3 ... week_index priority closed_relevance
741 1.197394 -18.726602 -5.305851 ... -33 Unbreak Now! True
6492 1.197394 -18.726602 -5.305851 ... 8 Medium False
6495 1.197394 -18.726602 -5.305851 ... 8 Medium False
8834 1.197394 -18.726602 -5.305851 ... -2 Medium False
9292 1.197394 -18.726602 -5.305851 ... -4 Medium True
9419 1.197394 -18.726602 -5.305851 ... -6 Medium NaN
10686 1.197394 -18.726602 -5.305851 ... 8 Low NaN
11301 1.197394 -18.726602 -5.305851 ... 2 Low True
11306 1.197394 -18.726602 -5.305851 ... 2 Low True
11312 1.197394 -18.726602 -5.305851 ... 2 Low True
[10 rows x 26 columns]
job finished, cleaning up
job pau at: Tue Sep 23 16:37:56 CDT 2025

195
p2/quest/parallel-mw-olmo-info-cat.log
View File

@ -1,195 +0,0 @@
setting up the environment by loading in conda environment at Thu Sep 4 18:05:51 CDT 2025
running the olmo labeling job at Thu Sep 4 18:05:52 CDT 2025
----------------------------------------
srun job start: Thu Sep 4 18:05:54 CDT 2025
Job ID: 3301934
Username: nws8519
Queue: gengpu
Account: p32852
----------------------------------------
The following variables are not
guaranteed to be the same in the
prologue and the job run script
----------------------------------------
PATH (in prologue) : /home/nws8519/.conda/envs/olmo/bin:/software/miniconda3/4.12.0/condabin:/home/nws8519/.local/bin:/home/nws8519/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/lpp/mmfs/bin:/hpc/usertools
WORKDIR is: /home/nws8519
----------------------------------------
Traceback (most recent call last):
File "/home/nws8519/.conda/envs/olmo/bin/accelerate", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py", line 50, in main
args.func(args)
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/accelerate/commands/launch.py", line 1222, in launch_command
multi_gpu_launcher(args)
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/accelerate/commands/launch.py", line 853, in multi_gpu_launcher
distrib_run.run(args)
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/run.py", line 883, in run
elastic_launch(
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 139, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
result = agent.run()
^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
result = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/agent/server/api.py", line 711, in run
result = self._invoke_run(role)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/agent/server/api.py", line 864, in _invoke_run
self._initialize_workers(self._worker_group)
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
result = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/agent/server/api.py", line 683, in _initialize_workers
self._rendezvous(worker_group)
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
result = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/agent/server/api.py", line 500, in _rendezvous
rdzv_info = spec.rdzv_handler.next_rendezvous()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/rendezvous/static_tcp_rendezvous.py", line 67, in next_rendezvous
self._store = TCPStore( # type: ignore[call-arg]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 29505, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use
Traceback (most recent call last):
File "/home/nws8519/.conda/envs/olmo/bin/accelerate", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py", line 50, in main
args.func(args)
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/accelerate/commands/launch.py", line 1222, in launch_command
multi_gpu_launcher(args)
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/accelerate/commands/launch.py", line 853, in multi_gpu_launcher
distrib_run.run(args)
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/run.py", line 883, in run
elastic_launch(
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 139, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
result = agent.run()
^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
result = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/agent/server/api.py", line 711, in run
result = self._invoke_run(role)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/agent/server/api.py", line 864, in _invoke_run
self._initialize_workers(self._worker_group)
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
result = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/agent/server/api.py", line 683, in _initialize_workers
self._rendezvous(worker_group)
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
result = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/agent/server/api.py", line 500, in _rendezvous
rdzv_info = spec.rdzv_handler.next_rendezvous()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/rendezvous/static_tcp_rendezvous.py", line 67, in next_rendezvous
self._store = TCPStore( # type: ignore[call-arg]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 29505, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use
srun: error: qgpu2005: task 0: Exited with exit code 1
srun: error: qgpu2008: task 2: Exited with exit code 1
[W904 18:21:24.281870443 socket.cpp:460] [c10d] waitForInput: poll for socket SocketImpl(fd=27, addr=[qgpu2005]:38060, remote=[qgpu2005]:29505) returned 0, likely a timeout
[W904 18:21:24.282308265 socket.cpp:485] [c10d] waitForInput: socket SocketImpl(fd=27, addr=[qgpu2005]:38060, remote=[qgpu2005]:29505) timed out after 900000ms
[W904 18:21:24.731952663 socket.cpp:460] [c10d] waitForInput: poll for socket SocketImpl(fd=27, addr=[qgpu2008]:35800, remote=[qgpu2005]:29505) returned 0, likely a timeout
[W904 18:21:24.733301968 socket.cpp:485] [c10d] waitForInput: socket SocketImpl(fd=27, addr=[qgpu2008]:35800, remote=[qgpu2005]:29505) timed out after 900000ms
Traceback (most recent call last):
File "/home/nws8519/.conda/envs/olmo/bin/accelerate", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py", line 50, in main
args.func(args)
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/accelerate/commands/launch.py", line 1222, in launch_command
multi_gpu_launcher(args)
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/accelerate/commands/launch.py", line 853, in multi_gpu_launcher
distrib_run.run(args)
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/run.py", line 883, in run
elastic_launch(
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 139, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
result = agent.run()
^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
result = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/agent/server/api.py", line 711, in run
result = self._invoke_run(role)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/agent/server/api.py", line 864, in _invoke_run
self._initialize_workers(self._worker_group)
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
result = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/agent/server/api.py", line 683, in _initialize_workers
self._rendezvous(worker_group)
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
result = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/agent/server/api.py", line 513, in _rendezvous
workers = self._assign_worker_ranks(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
result = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/agent/server/api.py", line 605, in _assign_worker_ranks
role_infos_bytes = store.multi_get(
^^^^^^^^^^^^^^^^
torch.distributed.DistStoreError: wait timeout after 900000ms, keys: /none/torchelastic/role_info/0, /none/torchelastic/role_info/1
Traceback (most recent call last):
File "/home/nws8519/.conda/envs/olmo/bin/accelerate", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py", line 50, in main
args.func(args)
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/accelerate/commands/launch.py", line 1222, in launch_command
multi_gpu_launcher(args)
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/accelerate/commands/launch.py", line 853, in multi_gpu_launcher
distrib_run.run(args)
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/run.py", line 883, in run
elastic_launch(
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 139, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
result = agent.run()
^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
result = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/agent/server/api.py", line 711, in run
result = self._invoke_run(role)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/agent/server/api.py", line 864, in _invoke_run
self._initialize_workers(self._worker_group)
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
result = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/agent/server/api.py", line 683, in _initialize_workers
self._rendezvous(worker_group)
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
result = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/agent/server/api.py", line 513, in _rendezvous
workers = self._assign_worker_ranks(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/metrics/api.py", line 138, in wrapper
result = f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/nws8519/.conda/envs/olmo/lib/python3.11/site-packages/torch/distributed/elastic/agent/server/api.py", line 605, in _assign_worker_ranks
role_infos_bytes = store.multi_get(
^^^^^^^^^^^^^^^^
torch.distributed.DistStoreError: wait timeout after 900000ms, keys: /none/torchelastic/role_info/0, /none/torchelastic/role_info/1
srun: error: qgpu2005: task 1: Exited with exit code 1
srun: error: qgpu2008: task 3: Exited with exit code 1
unsupervised olmo categorization pau at Thu Sep 4 18:21:24 CDT 2025

8
p2/quest/python_scripts/neurobiber_PCA.py
View File

@ -38,9 +38,9 @@ def format_df_data(df):
return x
if __name__ == "__main__":
biber_vec_df = pd.read_csv("/home/nws8519/git/mw-lifecycle-analysis/p2/quest/072525_pp_biberplus_labels.csv", low_memory=False)
biber_vec_df = pd.read_csv("/home/nws8519/git/mw-lifecycle-analysis/p2/quest/092325_biberplus_complete_labels.csv", low_memory=False)
biber_vec_df = biber_vec_df[biber_vec_df['comment_type'] == 'task_description']
biber_vec_df = biber_vec_df[biber_vec_df['AuthorPHID'] != "PHID-USER-idceizaw6elwiwm5xshb"]
#biber_vec_df = biber_vec_df[biber_vec_df['AuthorPHID'] != "PHID-USER-idceizaw6elwiwm5xshb"]
#biber_vec_df = biber_vec_df[biber_vec_df['comment_text'] != 'nan']
biber_vecs = format_df_data(biber_vec_df)
#handoff to PCA model
@ -56,7 +56,7 @@ if __name__ == "__main__":
'''
pca = PCA(n_components=18)
biber_vecs_pca = pca.fit_transform(biber_vecs)
with open('092325_pca.pkl', 'wb') as f:
with open('092525_description_pca.pkl', 'wb') as f:
pickle.dump(pca, f)
selected_axis = "AuthorWMFAffil"
@ -121,5 +121,5 @@ if __name__ == "__main__":
plt.legend(title=selected_axis, bbox_to_anchor=(1.05, 1), loc=2)
'''
g.fig.tight_layout()
g.savefig(f"description_{selected_axis}_092325_biber_pca_final.png", dpi=300)
g.savefig(f"description_{selected_axis}_092525_biber_pca_final.png", dpi=300)
plt.show()

10
p2/quest/sampling-mw-olmo-info-cat.log
View File

@ -1,10 +0,0 @@
setting up the environment by loading in conda environment at Mon Sep 22 20:07:58 CDT 2025
running the sampling job at Mon Sep 22 20:07:58 CDT 2025
[nltk_data] Downloading package punkt_tab to
[nltk_data] /home/nws8519/nltk_data...
[nltk_data] Package punkt_tab is already up-to-date!
cuda
NVIDIA A100-SXM4-80GB
_CudaDeviceProperties(name='NVIDIA A100-SXM4-80GB', major=8, minor=0, total_memory=81153MB, multi_processor_count=108, uuid=290bcddd-9b2f-3a5b-4cbd-b17d9ec05044, L2_cache_size=40MB)
Loading checkpoint shards: 0%| | 0/12 [00:00<?, ?it/s] Loading checkpoint shards: 8%|▊ | 1/12 [00:00<00:04, 2.32it/s] Loading checkpoint shards: 17%|█▋ | 2/12 [00:01<00:06, 1.64it/s] Loading checkpoint shards: 25%|██▌ | 3/12 [00:01<00:05, 1.66it/s] Loading checkpoint shards: 33%|███▎ | 4/12 [00:02<00:04, 1.64it/s] Loading checkpoint shards: 42%|████▏ | 5/12 [00:02<00:04, 1.68it/s] Loading checkpoint shards: 50%|█████ | 6/12 [00:03<00:03, 1.60it/s] Loading checkpoint shards: 58%|█████▊ | 7/12 [00:04<00:03, 1.65it/s] Loading checkpoint shards: 67%|██████▋ | 8/12 [00:04<00:02, 1.59it/s] Loading checkpoint shards: 75%|███████▌ | 9/12 [00:05<00:01, 1.56it/s] Loading checkpoint shards: 83%|████████▎ | 10/12 [00:06<00:01, 1.64it/s] Loading checkpoint shards: 92%|█████████▏| 11/12 [00:06<00:00, 1.76it/s] Loading checkpoint shards: 100%|██████████| 12/12 [00:06<00:00, 1.82it/s]
sampling pau at Mon Sep 22 20:11:48 CDT 2025

2
p2/quest/slurm_jobs/pca_run.sh
View File

@ -8,7 +8,7 @@
#SBATCH --mem=64G
#SBATCH --cpus-per-task=4
#SBATCH --job-name=neurobiber-pca
#SBATCH --output=neurobiber-pca.log
#SBATCH --output=092525_neurobiber-pca.log
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH --mail-user=gaughan@u.northwestern.edu

BIN
p2/quest/subcomment_AuthorWMFAffil_090425_biber_pca_final.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 2.1 MiB