collective/wikiq
20
0
Fork 0
Code Issues Pull Requests 3 Activity

add spark program for running group by users

Browse Source
This commit is contained in:
Nathan TeBlunthuis
2018-08-31 20:40:22 +00:00
parent ff689c71dd
commit bc1f5428f0
3 changed files with 111 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
wikiq_users/run_wikiq_users_cluster.sh Executable file
View File

@@ -0,0 +1,2 @@
#!/usr/bin/env bash
spark-submit --master spark://n0649:18899 wikiq_users_spark.py --output-format parquet -i "/com/output/wikiq-enwiki-20180301/enwiki-20180301-pages-meta-history*.tsv" -o "/com/output/wikiq-users-enwiki-20180301-parquet/" --num-partitions 500