Chapter 5: Distributed Memory Programming with MPI

The MPI Model

• An industrial standard to support distributed memory programming
• Multiple implementations
  • MPICH
  • OpenMPI
• Components
  • compiler wrapper mpicc
  • library mpi.h
  • runner mpiexec or mpirun (not official)

Basic MPI Concepts

• Communicator - collection of processes that can communicate through MPI
• Process - the running unit in MPI, same or different computation nodes
• Size $p$ - number of processes in a communicator
• Rank $0, 1, \cdots p-1$ - unique identifier of a process in a communicator
• API types
  • peer-to-peer (P2P)
  • collective
• Communication call types
  • blocking
  • non-blocking
• Setup/Finalize
  • MPI_Init
  • MPI_Finalize
• Data types
• Message matching

Synchronization

• MPI_Barrier

Peer-to-peer APIs

• Send/receive
• Send and receive in one call

Collective APIs

• Map
• Reduce
• All Reduce
• Broadcast
• Gather/Scatter
• All Gether

Derived Data-Type

• Customize data type

Evaluation

• MPI_Wtime
• MPI_Barrier

Safety

• Buffering vs Blocking
• Deadlock