PySpark Environment For Learning

PySpark Environment

Spark can be configured in local or cluster mode. It also supports standalone, YARN, and Kubernetes as resource managers. Installation of Spark in cluster mode requires a lot of experience and effort. It is not necessary in this course.

For these teaching examples, run PySpark in local mode inside a Google Colab notebook. The notebook environment hosts the Spark process on one machine. Spark’s Standalone cluster manager is a separate deployment option; the local notebook setup does not create a Standalone cluster.

Google Colab

Google Colaboratory (also known as Google Colab or simply Colab) is a cloud-based platform that allows users to write, run, and share Python code using Jupyter notebooks. It is a free service provided by Google that allows users to access powerful computing resources, including GPUs and TPUs, without the need for expensive hardware.

With Colab, you can easily create and share documents that contain live code, equations, visualizations, and text. You can also collaborate with others in real-time, making it an ideal tool for data science teams, researchers, educators, and students.

Google Colab website

Tutorials for Google Colab:

• Google Colab Tutorial for Beginners (towardai.net)
• Google Colab Tutorial (tutorialspoint.com)

Install PySpark on Colab

Input %pip install pyspark in a code cell and run.

Access PySpark functionalities

• Import PySpark components in a code cell
• Create a SparkSession object (usually named spark) and use it as the primary entry point to PySpark functionalities
• Access SparkContext through spark.sparkContext when you need lower-level RDD operations

File Operations

• Each notebook runs in a separate virtual machine with its own file system.
• This file system is temporary and will be deleted when the virtual machine is recycled.
• To load data (three options)
  1. write code to download data from a URL
  2. upload data from your local machine
  3. mount Google Drive to the virtual machine
• To persist data (two options)
  1. save data to Google Drive
  2. download data to your local machine using the file explorer sidebar

Export the notebook

Use the menu “File -> print” to generate a PDF file or print it to a printer.

Save Data

You can upload and download in the file explorer sidebar or mount your Google Drive to the Colab virtual machine.