PySpark Programming for Big Data

Spark supports multiple programming languages including Java, Scala, Python, and R. In this course, we will use Python as the programming language to work with Spark. The PySpark components are built on top of Spark Core and provide Python APIs to interact with Spark.

We will only briefly introduce how to program in PySpark to manipulate datasets and achieve certain big data processing tasks.

Why PySpark

• Other languages

  • Scala

    • native language for Spark

    • best performance

    • concise and powerful language

    • the language itself is not worth learning as a new language compared to Python

    • complicated syntax

  • Java: too verbose

  • R: not a general-purpose language

• Advantages

  • Easier to install using pip than Scala

  • You may have learned Python already

  • Python worth the time to learn

PySpark Basics

• Concepts

  SparkContext

  Low-level context for RDD operations. In modern PySpark code, you usually access it from spark.sparkContext.

  SparkSession

  Primary entry point to PySpark, including DataFrame and Spark SQL APIs.

  RDD (Resilient Distributed Dataset)

  A fundamental building block of PySpark which is fault-tolerant, immutable distributed collection of objects

  DataFrame

  A distributed collection of data organized into named columns.

  Dataset API

  A strongly-typed API available in Scala/Java. In PySpark, DataFrame is the main structured API.

• RDD

  • from Spark Core module

  • lower-level API

  • for unstructured data and semi-structured data

• DataFrame (and Dataset API in Scala/Java)

  • from Spark SQL module

  • higher-level API

  • for structured data

Related Python Concepts

As the programming language used in PySpark is Python, we will need to know some Python concepts to work with PySpark. Please refer to the Python programming resources online if you are not familiar with Python.

We will learn some useful python concepts to be used in data manipulation using PySpark APIs. For details refer to any Python programming resources online.

Suggested learning resources

• G4G Python Tutorial

• Official Python Reference

• Ask Chatbot on specific questions: ChatGPT, Google Gemini, Claude AI, etc.

Python Basics

• Python is a high-level, interpreted, interactive, and object-oriented scripting language.

• Data types

  • int

  • float

  • str

  • tuple - immutable list of objects of any type

  • list - mutable list of objects of any type

  • dict - key-value pair of objects of any type

• Control structures

  • if-elif-else

  • for loop

  • while loop

• Functions

  • def keyword

  • return keyword

  • positional arguments

  • keyword arguments

  • default arguments

  • lambda functions

Lambda Functions

• A simple and disposable way to write a python function.

• Many RDD transformations require a function as its parameter

• Syntax

  • lambda <input list>: <return value>

  • start with lambda keyword

  • followed by a list of parameters

  • after the colon, provide the return value calculated based on the parameters

• Examples

  • lambda x: x*x, square the input

  • lambda line: line.split(), split a text to a list of text using spaces as delimiters

  • lambda paragraph: paragraph.split('\n'), split a text to a list of lines using new line character as delimiters

  • lambda word: (word, 1), convert a work to a key-value pair of word and 1

  • lambda word: len(word), convert a string to its length

  • lambda pair: pair[0], return the first element of the pair

  • lambda pair: pair[1], return the second element of the pair

  • lambda x, y: x+y, sum up two value

  • lambda p1, p2: p1[1] + p2[1], sum up the second values in two pairs