Module 5: Distributed Memory Programming with MPI

The MPI Model

  • An industrial standard to support distributed memory programming

  • Multiple implementations

    • MPICH

    • OpenMPI

  • Components

    • compiler wrapper mpicc

    • library mpi.h

    • runner mpiexec or mpirun (not official)

Basic MPI Concepts

  • Communicator - collection of processes that can communicate through MPI

  • Process - the running unit in MPI, same or different computation nodes

  • Size \(p\) - number of processes in a communicator

  • Rank \(0, 1, \cdots p-1\) - unique identifier of a process in a communicator

  • API types

    • peer-to-peer (P2P)

    • collective

  • Communication call types

    • blocking

    • non-blocking

  • Setup/Finalize

    • MPI_Init

    • MPI_Finalize

  • Data types

  • Message matching

Synchronization

  • MPI_Barrier

Peer-to-peer APIs

  • Send/receive

  • Send and receive in one call

Collective APIs

  • Map

  • Reduce

  • All Reduce

  • Broadcast

  • Gather/Scatter

  • All Gether

Derived Data-Type

  • Customize data type

Evaluation

  • MPI_Wtime

  • MPI_Barrier

Safety

  • Buffering vs Blocking

  • Deadlock