.. highlight:: c++
  :linenothreshold: 5

*************************************************
Module 5: Distributed Memory Programming with MPI
*************************************************

The MPI Model
=============
+ An industrial standard to support distributed memory programming
+ Multiple implementations

  * MPICH
  * OpenMPI

+ Components

  * compiler wrapper ``mpicc``
  * library ``mpi.h``
  * runner ``mpiexec`` or ``mpirun`` (not official)

Basic MPI Concepts
==================
+ Communicator - collection of processes that can communicate through MPI
+ Process - the running unit in MPI, same or different computation nodes
+ Size :math:`p` - number of processes in a communicator
+ Rank :math:`0, 1, \cdots p-1` - unique identifier of a process in a
  communicator
+ API types

  * peer-to-peer (P2P)
  * collective

+ Communication call types

  * blocking
  * non-blocking

+ Setup/Finalize

  * MPI_Init
  * MPI_Finalize

+ Data types
+ Message matching

Synchronization
===============
+ MPI_Barrier

Peer-to-peer APIs
=================
+ Send/receive
+ Send and receive in one call

Collective APIs
===============
+ Map
+ Reduce
+ All Reduce
+ Broadcast
+ Gather/Scatter
+ All Gether

Derived Data-Type
=================
+ Customize data type

Evaluation
==========
+ MPI_Wtime
+ MPI_Barrier

Safety
======
+ Buffering vs Blocking
+ Deadlock