groceryheist/cdsc_reddit
1
0
Fork 0
Code Activity

Rename spark script to reflect that it is for comments.

Browse Source
This commit is contained in:
Nate E TeBlunthuis
2020-07-03 14:00:36 -07:00
parent 94c7a74bd9
commit 6dca79a41f
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
reddit_bz2_2parquet.py → comments_2_parquet.py
View File

@@ -6,7 +6,7 @@ from pyspark.sql.types import *
from pyspark import SparkConf, SparkContext
from pyspark.sql import SparkSession, SQLContext
conf = SparkConf().setAppName("Reddit to bz2")
conf = SparkConf().setAppName("Reddit comments to parquet")
conf = conf.set('spark.sql.crossJoin.enabled',"true")
spark = SparkSession.builder.getOrCreate()