UNITRANS: Simplifying parallel (re-)decompositions in ScalES

Dr. Mathias Pütz, IBM
This email address is being protected from spambots. You need JavaScript enabled to view it.

(slides)

Earth-System-Models usually consist of multiple, complex modules covering different physical aspects such as air & ocean circulation, tracer transport, radiation, clouds & precipitation, tracer chemistry and biosphere interaction. Each of those sub-models can have very different computations load distributions requiring special decompositions to obtain satisfactory load balance for parallelization. To efficiently address the problem of re-shuffling data from one decomposition into another we created a simple, yet powerful middleware communication library named UNITRANS, which allows the model developers to define logical decompositions and physical data representations separately without relying on the capabilities of a specific low level communication library. Decompositions can be changed independently from the local process data layout and thus without touching a single line of sub-model code. The developers are no longer required to actually write code to implement the different transpositions themselves; finding the best way to transpose data from data a data layout in one decomposition to another is handled by the UNITRANS library.

We will demonstrate, how we leveraged the UNITRANS libary to increase the efficiency of the ECHAM and MESSY ESM, both with respect to better load balancing and improved performance of the actual communication steps themselves. UNITRANS will be available under an open-source license from IBM and we plan some additional enhancements in the future such as: implementations
on top of low-level, high-performance communication protocols such as IBM PAMI, providing C/C++ language bindings (currently F90 only) and automatic use of collective operations to accelerate certain types of transpositions.