re-done total pca

2025-10-20 12:38:44 -05:00 · 2025-10-20 12:38:44 -05:00 · f146016eac
commit f146016eac
parent 2e8b85d3e9
4 changed files with 146357 additions and 1 deletions
--- a/p2/quest/102025_total_neurobiber-pca.log
+++ b/p2/quest/102025_total_neurobiber-pca.log
@ -0,0 +1,341 @@
+starting the job at: Mon Oct 20 12:35:32 CDT 2025
+setting up the environment
+running the neurobiber labeling script
+0        [VisualEditor: Editing at top of page shows pa...
+1        [Change 86685 merged by jenkins-bot:\nFollow-u...
+2        [*** Bug 54785 has been marked as a duplicate ...
+3        [Change 86685 had a related patch set uploaded...
+4        [Don't release any new VE updates for three mo...
+                               ...                        
+25023    [Could you attach a screenshot please?, Drag &...
+25024    [2 login-links on page but different behaviour...
+25025    [Sorry for not reply-ing., I did a test and co...
+25026                        [SCREEN_NAME: Please answer.]
+25027    [I cannot replicate this., What's the name of ...
+Name: olmo_cleaned_sentences, Length: 25028, dtype: object
+[[13. ]
+ [18. ]
+ [ 6.5]
+ ...
+ [ 5.5]
+ [ 3. ]
+ [ 6. ]]
+Number of PCs explaining 90% variance: 23
+Variance of each PCA component: [246.61008055 133.36698159  74.94326075  61.33534517  54.80370961
+  35.82933802  29.27972058  23.85842265  19.90839023  17.25654085
+  14.9325249   14.69222654  12.40988952  10.49363109   9.72451692
+   8.38950194   8.11897062   7.49040269   7.26770791   6.45798815
+   6.22392084   5.87412538   5.70894249]
+PC1:
+normalized_CAP: 0.666
+normalized_NNP: 0.595
+median_sentence_length: -0.315
+normalized_DET: -0.141
+normalized_PREP: -0.119
+normalized_PIN: -0.119
+normalized_ART: -0.090
+normalized_VPRT: -0.079
+normalized_RB: -0.075
+normalized_PRP: -0.069
+PC2:
+median_sentence_length: 0.930
+normalized_NNP: 0.313
+normalized_CAP: 0.093
+normalized_DET: -0.065
+normalized_VPRT: -0.065
+normalized_RB: -0.065
+normalized_AUXB: -0.050
+normalized_PRP: -0.040
+normalized_SBJP: -0.040
+normalized_X: 0.034
+PC3:
+normalized_NN: 0.759
+normalized_NNP: -0.288
+normalized_RB: -0.261
+normalized_PRP: -0.228
+normalized_SBJP: -0.228
+normalized_CAP: 0.210
+normalized_VPRT: -0.164
+normalized_FPP1: -0.115
+normalized_NUM: 0.102
+normalized_INF: -0.096
+PC4:
+normalized_CAP: 0.526
+normalized_PIN: 0.481
+normalized_PREP: 0.481
+normalized_NNP: -0.212
+normalized_PRP: 0.185
+normalized_SBJP: 0.185
+normalized_X: -0.141
+normalized_RB: 0.133
+normalized_CONJ: 0.129
+normalized_INF: 0.125
+PC5:
+normalized_NNP: 0.478
+normalized_PREP: 0.454
+normalized_PIN: 0.454
+normalized_CAP: -0.416
+normalized_RB: -0.267
+normalized_SBJP: -0.139
+normalized_PRP: -0.139
+median_sentence_length: -0.137
+normalized_VPRT: -0.108
+normalized_CONJ: 0.107
+PC6:
+normalized_DET: 0.615
+normalized_ART: 0.387
+normalized_NN: 0.289
+normalized_NNP: 0.273
+normalized_VPRT: 0.260
+normalized_X: -0.253
+normalized_AUXB: 0.209
+normalized_NUM: -0.172
+normalized_INDA: 0.159
+normalized_INF: -0.156
+PC7:
+normalized_NN: 0.477
+normalized_PRP: 0.451
+normalized_SBJP: 0.451
+normalized_NNP: 0.249
+normalized_FPP1: 0.231
+normalized_DET: -0.223
+normalized_CAP: -0.176
+normalized_AUXB: -0.169
+normalized_PASS: -0.134
+normalized_PIT: 0.125
+PC8:
+normalized_RB: 0.781
+normalized_NN: 0.240
+normalized_PRP: -0.207
+normalized_SBJP: -0.206
+normalized_DET: -0.185
+normalized_JJ: -0.171
+normalized_ART: -0.147
+normalized_NNP: 0.143
+normalized_TIME: 0.140
+normalized_X: -0.137
+PC9:
+normalized_JJ: 0.676
+normalized_INF: 0.326
+normalized_VPRT: -0.300
+normalized_NUM: -0.258
+normalized_PASS: -0.218
+normalized_AUXB: -0.216
+normalized_ART: 0.200
+sentence_count: 0.153
+normalized_RB: 0.136
+normalized_CONJ: -0.134
+PC10:
+normalized_INF: 0.647
+normalized_JJ: -0.513
+normalized_VPRT: -0.334
+normalized_DET: 0.260
+normalized_ART: 0.158
+normalized_TO: 0.127
+normalized_PRIV: 0.101
+normalized_NUM: 0.100
+normalized_AUXB: -0.094
+normalized_INDA: 0.077
+PC11:
+sentence_count: 0.554
+normalized_INF: -0.400
+normalized_AUXB: -0.331
+normalized_VPRT: -0.304
+normalized_JJ: -0.301
+normalized_ART: 0.208
+normalized_RB: 0.169
+normalized_PASS: -0.153
+normalized_BEMA: -0.129
+normalized_TO: -0.127
+PC12:
+sentence_count: 0.673
+normalized_X: -0.380
+normalized_RB: -0.235
+normalized_VBD: -0.235
+normalized_INF: 0.214
+normalized_VPRT: 0.188
+normalized_AUXB: 0.182
+normalized_DET: -0.153
+normalized_NUM: -0.150
+normalized_ART: -0.148
+PC13:
+normalized_X: 0.773
+normalized_VPRT: 0.284
+normalized_VBD: -0.259
+normalized_PUBV: -0.253
+sentence_count: 0.210
+normalized_NUM: -0.180
+normalized_CONJ: -0.144
+normalized_NOMZ: -0.116
+normalized_INF: 0.103
+normalized_UH: -0.096
+PC14:
+normalized_NUM: 0.789
+sentence_count: 0.222
+normalized_QUOT: -0.181
+normalized_JJ: 0.174
+normalized_VBD: -0.171
+normalized_VPRT: 0.165
+normalized_UH: -0.160
+normalized_PGAS: -0.156
+normalized_RB: 0.153
+normalized_INDA: 0.126
+PC15:
+normalized_VBD: 0.495
+normalized_AUXB: 0.374
+normalized_QUOT: -0.369
+normalized_CONT: -0.288
+sentence_count: 0.269
+normalized_PASS: 0.261
+normalized_VPRT: -0.244
+normalized_X: 0.174
+normalized_NOMZ: -0.154
+normalized_PUBV: 0.142
+PC16:
+normalized_QUOT: 0.594
+normalized_CONT: 0.441
+normalized_VBD: 0.400
+normalized_UH: -0.278
+normalized_NUM: 0.167
+normalized_PUBV: 0.164
+normalized_CONJ: -0.164
+normalized_NOMZ: -0.156
+normalized_STPR: 0.127
+normalized_AUXB: 0.121
+PC17:
+normalized_PUBV: 0.504
+normalized_CONJ: -0.473
+normalized_PGAS: 0.303
+normalized_VPRT: 0.255
+normalized_NOMZ: 0.245
+normalized_PASS: -0.219
+normalized_QUOT: -0.202
+normalized_CONT: -0.160
+normalized_AUXB: -0.128
+normalized_UH: -0.123
+PC18:
+normalized_CONJ: 0.612
+normalized_PUBV: 0.504
+normalized_UH: -0.308
+normalized_NUM: -0.210
+normalized_NOMZ: 0.166
+normalized_PGAS: -0.164
+normalized_ART: 0.156
+normalized_X: 0.155
+normalized_VPRT: 0.152
+normalized_DEMP: -0.116
+PC19:
+normalized_UH: 0.727
+normalized_PGAS: -0.434
+normalized_PUBV: 0.252
+normalized_NOMZ: -0.184
+normalized_VPRT: 0.182
+normalized_CCONJ: -0.171
+normalized_VBD: 0.142
+normalized_TIME: -0.092
+normalized_ART: 0.091
+normalized_INDA: 0.083
+PC20:
+normalized_ART: 0.432
+normalized_DET: -0.341
+normalized_DEMO: -0.286
+normalized_INDA: 0.280
+normalized_DEMP: -0.273
+normalized_CONJ: -0.271
+normalized_NOMZ: 0.216
+normalized_PIT: 0.211
+normalized_FPP1: -0.199
+normalized_AUXB: 0.199
+PC21:
+normalized_PGAS: 0.725
+normalized_UH: 0.307
+normalized_AUXB: 0.253
+normalized_NOMZ: -0.222
+normalized_CONJ: 0.211
+normalized_ART: 0.154
+normalized_BEMA: 0.143
+normalized_NUM: 0.110
+normalized_CCONJ: -0.106
+normalized_QUES: -0.106
+PC22:
+normalized_CCONJ: 0.625
+normalized_NOMZ: -0.330
+normalized_QUES: -0.312
+normalized_VPRT: 0.243
+normalized_AUXB: -0.234
+normalized_PRIV: 0.176
+normalized_BEMA: -0.175
+normalized_TIME: 0.174
+normalized_SCONJ: -0.162
+normalized_WH: -0.132
+PC23:
+normalized_PRIV: 0.614
+normalized_NOMZ: 0.395
+normalized_VBD: 0.269
+normalized_CCONJ: -0.257
+normalized_QUES: -0.246
+normalized_PUBV: -0.241
+normalized_FPP1: 0.217
+normalized_PIT: -0.154
+normalized_TO: -0.119
+normalized_CONJ: 0.117
+Top 10 PC1 values:
+              PC1        PC2  ...  date_created     comment_type
+23531  122.614493  26.508184  ...    1424754141  task_subcomment
+707    122.597764  26.498206  ...    1363132183  task_subcomment
+744    122.597764  26.498206  ...    1353551242  task_subcomment
+749    122.597764  26.498206  ...    1353384355  task_subcomment
+2243   122.597764  26.498206  ...    1356175107  task_subcomment
+5921   122.597764  26.498206  ...    1353366778  task_subcomment
+5933   122.597764  26.498206  ...    1353123761  task_subcomment
+5935   122.597764  26.498206  ...    1353386649  task_subcomment
+10080  122.597764  26.498206  ...    1366298361  task_subcomment
+10418  122.597764  26.498206  ...    1355363288  task_subcomment
+
+[10 rows x 34 columns]
+
+Bottom 10 PC1 values:
+              PC1         PC2  ...  date_created      comment_type
+16080 -172.850818  487.621918  ...    1350678600  task_description
+24812 -145.955594  438.167891  ...    1463441072   task_subcomment
+24813 -145.770728  438.255988  ...    1463441050   task_subcomment
+13983  -97.131561  274.630845  ...    1380947348   task_subcomment
+16510  -92.274843  294.480228  ...    1354470131   task_subcomment
+161    -74.924301  196.826216  ...    1374730027   task_subcomment
+22005  -66.493946  211.966534  ...    1440633395   task_subcomment
+24815  -66.161789  174.725549  ...    1463439992   task_subcomment
+6163   -65.995800  195.974595  ...    1379611711   task_subcomment
+20858  -58.834964  191.798984  ...    1441031208   task_subcomment
+
+[10 rows x 34 columns]
+Top 10 PC2 values:
+              PC1         PC2  ...  date_created      comment_type
+16080 -172.850818  487.621918  ...    1350678600  task_description
+24813 -145.770728  438.255988  ...    1463441050   task_subcomment
+24812 -145.955594  438.167891  ...    1463441072   task_subcomment
+16510  -92.274843  294.480228  ...    1354470131   task_subcomment
+13983  -97.131561  274.630845  ...    1380947348   task_subcomment
+22005  -66.493946  211.966534  ...    1440633395   task_subcomment
+161    -74.924301  196.826216  ...    1374730027   task_subcomment
+6163   -65.995800  195.974595  ...    1379611711   task_subcomment
+20858  -58.834964  191.798984  ...    1441031208   task_subcomment
+24815  -66.161789  174.725549  ...    1463439992   task_subcomment
+
+[10 rows x 34 columns]
+
+Bottom 10 PC2 values:
+             PC1        PC2  ...  date_created     comment_type
+17259 -11.502462 -20.654068  ...    1321220595  task_subcomment
+24780  -7.649212 -18.480106  ...    1420344576  task_subcomment
+22246   3.193427 -17.554603  ...    1461480989  task_subcomment
+20534  -7.911332 -16.886866  ...    1443534834  task_subcomment
+21315 -10.345515 -16.806925  ...    1453975581  task_subcomment
+5078  -14.656633 -16.709304  ...    1377104818  task_subcomment
+15192 -14.489305 -16.685012  ...    1361930342  task_subcomment
+20939   0.546084 -16.640121  ...    1436977570  task_subcomment
+7055   -0.820605 -16.571093  ...    1377020909  task_subcomment
+7471   -0.088132 -16.477148  ...    1386166246  task_subcomment
+
+[10 rows x 34 columns]
+job finished, cleaning up
+job pau at: Mon Oct 20 12:36:03 CDT 2025
--- a/p2/quest/102025_total_pca.pkl
+++ b/p2/quest/102025_total_pca.pkl
--- a/p2/quest/102025_total_pca_df.csv
+++ b/p2/quest/102025_total_pca_df.csv
--- a/p2/quest/python_scripts/neurobiber_PCA.py
+++ b/p2/quest/python_scripts/neurobiber_PCA.py
@ -131,7 +131,8 @@ if __name__ == "__main__":
    pc_dict['resolution_outcome'] = biber_vec_df['resolution_outcome']
    pc_dict['TaskPHID'] = biber_vec_df['TaskPHID']
    pc_dict['AuthorPHID'] = biber_vec_df['AuthorPHID']
-    pc_dict['date_created'] = biber_vec_df['date_created'] 
+    pc_dict['date_created'] = biber_vec_df['date_created']
+    pc_dict['comment_type'] = biber_vec_df['comment_type'] 


    plot_df = pd.DataFrame(pc_dict)