Message Passing Interface (MPI) is the message-passing library most widely used to provide communications in clusters. There are several MPI implementations like MPICH, CHIMP, LAM, OPEN MPI, etc. We have developed a library called PRAcTICaL-MPI (PoRtable AdpaTIve Compression Library) that reduces the data volume by using loss-less compression among processes. Furthermore, PRAcTICaL-MPI allows turn compression on and off and select at run-time the most appropriate compression algorithm depending on the characteristics of each message, network performance, and compression algorithms behavior.
PRAcTICaL-MPI is developed over PMPI, that it is the MPI standard profiling interface. This interface captures the MPI calls made by each program. We have used PMPI to customize MPI behavior, and include all our adaptive compression techniques. One of the advantages of implementing our compression techniques over PMPI is that is not need to modify the source code of the MPI implementations and the applications. This is due to PMPI allows selective replacement of MPI routines at link time.
We have tested the PRAcTICaL-MPI with MPICH2, with different MPI applications, and also with compression algorithms like LZO, RLE, FPC , etc. The evaluations shows that compressing the MPI messages with the proper algorithm and only when is it worth it, we get in most of the cases a great reduction in the overall execution time of MPI applications.
The challenge now is to update PRAcTICaL-MPI, with new and faster compression algorithms like Snappy or PFOR. Furthermore, we want to evaluate the PRAcTICaL-MPI with OPEN MPI and X-MPI by using HECToR and ECDF clusters.