PySpark Programming: RDD

RDD (Resilient Distributed Dataset)

• External readings

  • RDD Programming Guide
  • A Comprehensive Guide to PySpark RDD Operations

• The low-level abstraction in PySpark

• Can do anything with RDDs but the DataFrame API is more user-friendly and efficient for structured data

• RDDs are more flexible and can be used for unstructured and semi-structured data

• Characteristics

  • Distributed (data is distributed across multiple machines)
  • Immutable (once created, it cannot be changed)
  • Lazy evaluation: transformations record work to perform; an action triggers the computation needed for its result. A shuffle is a possible data redistribution step, not the trigger for execution.

• Two types of operations

  • As extensions to traditional map/reduce paradigm
  • Transformations
    • Takes an RDD and return another RDD
    • Distributed RDD to distributed RDD
    • Can be chained
    • Include but more than just mapping
    • Transformations are lazy
    • Some transformations require a shuffle when their results are computed
  • Actions
    • Trigger computation on an RDD
    • Some return results to the driver; others write output or perform side effects
    • Include but more than just reducing
    • All actions will trigger the execution of the DAG

• Useful transformations

  • map(func), apply a function to every element in an iterable, func should transform one item to another
  • flatMap(func), apply a function to every element in an iterable and flatten the resulting list, func should take one item and generate a list
  • filter(func), filter elements according to the returned value of a boolean function applied to every element, func should return True or False according to the input

• Useful actions

  • collect(), collect all data and return a local list, Never use with large dataset
  • count()
  • first()
  • take(num), take the first num of values and return a local list
  • reduce(func), func should take two parameters and give one result
  • sum()
  • max()
  • min()
  • mean()
  • countByValue()
    • Python dict like type as output {value1: count1, value2: count2, ...}
    • models key-value pair
    • key must be unique

• Pair RDD

  A pair RDD is an RDD whose elements are key-value pairs. PySpark provides key-based operations that use the first element of each pair as its key.

  • example [(key1, value1), (key2, value2), ...]
  • Useful transformations for pair RDDs
    • sortByKey(ascending=True)
    • sortBy(func, ascending=True)
      • func should take an item and return the value used to perform sorting
      • e.g. pairs.sortBy(lambda p: p[1]) use the value (second item in the pair) to sort the pairs
    • groupByKey()
      • create a new RDD with each item being pairs with a same key
    • reduceByKey(func)
      • Combine values with the same key and return a new pair RDD. This is a transformation.
  • Useful actions for pair RDDs
    • countByKey()

For the transformation/action distinction and shuffle behavior, see the Spark RDD programming guide.