mgaughan/mw-convo-collections
1
0
Fork 0
Code

getting kibo set for data collection

Browse Source
This commit is contained in:
Matthew Gaughan 2024-12-11 17:20:26 -06:00
parent b0dfcf3a13
commit 6cb3296e8e
1 changed files with 6 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
src/lib/get_wiki_activity.py
View File

@ -4,6 +4,12 @@ import re
import json
import pandas as pd
import findspark
findspark.init("/home/SOC.NORTHWESTERN.EDU/nws8519/spark-3.2.0-bin-hadoop3.2")
from pyspark.sql import SparkSession
from pyspark.sql import functions as F, types as T, Window
#breaking
import wmfdata.spark as wmfspark
# TODO Get a list of bots in the project
# TODO get all mws wikis