collective/cdsc_reddit
13
0
Fork 0
Code

Rename spark script to reflect that it is for comments.

Browse Source
This commit is contained in:
Nate E TeBlunthuis 2020-07-03 14:00:36 -07:00
parent 94c7a74bd9
commit 6dca79a41f
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
reddit_bz2_2parquet.py → comments_2_parquet.py
View File

@ -6,7 +6,7 @@ from pyspark.sql.types import *
from pyspark import SparkConf, SparkContext from pyspark import SparkConf, SparkContext
from pyspark.sql import SparkSession, SQLContext from pyspark.sql import SparkSession, SQLContext
conf = SparkConf().setAppName("Reddit to bz2") conf = SparkConf().setAppName("Reddit comments to parquet")
conf = conf.set('spark.sql.crossJoin.enabled',"true") conf = conf.set('spark.sql.crossJoin.enabled',"true")
spark = SparkSession.builder.getOrCreate() spark = SparkSession.builder.getOrCreate()