Chapter 5: PySpark Programming

In the previous chapter, we have introduced the data processing techniques and tools. In this chapter, we will introduce PySpark, the Python API for Spark. We will cover the basics of PySpark programming, including RDDs and DataFrames, and provide examples of how to use PySpark for big data processing tasks.

Contents:

• PySpark Environment For Learning
  • PySpark Environment
  • Google Colab
• PySpark Programming for Big Data
  • Why PySpark
  • PySpark Basics
  • Related Python Concepts
• PySpark Programming: RDD
  • RDD (Resilient Distributed Dataset)
• PySpark Programming: Spark SQL and DataFrame API
  • Outline
  • Spark SQL Overview
  • Performance optimizations
  • Applications of Spark SQL
  • DataFrame

Notebooks for practice

• Download PySpark Basics and RDD Notebook
• Download PySpark DataFrame Notebook