-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add spark example to Sample Notebook #1003
Conversation
add notebook description for spark session
typo fix
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi Kristin. Please take a look at my comments. Cheers
" .appName(\"Spark Session for Bike Sharing Data\") \\\n", | ||
" .getOrCreate()\n", | ||
"\n", | ||
"path=\"gs://kristin_serverless_pyspark/bikesharingdemand-train.csv\"\n", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this csv too big? If it is small is better to have it under a resources folder and not hardcode a GCS path that may not exist in the future
"This notebook let users \n", | ||
"* run a sample Spark Session with a sample CSV file\n", | ||
"* verify if GCS buckets are properly mounted as a file system and\n", | ||
"* execute Python files that are stored in mounted GCS buckets by !python and %run commands and\n" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to keep this part of the implementation? I mean the gcsfuse and running a python file from the notebook. I thought that you were going to create a separate example to demonstrate that, and keep this project running only PySpark
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kristin-kim If you can address the review questions, we can try to get this merged.
Closed as stale. Please re-open if I'm wrong. |
Add a simple spark example to existing sample Notebook of folder