Parallel Implementation and Optimization of Hybrid-Grid FEM-based Ground Motion Simulation Tool in Large Multicore Cluster Computer

Abstract

This paper presents the implementation and optimization of hybrid parallel programming model to an FEM-based simulation tool for earthquake wave propagation. A prepartitioning method is implemented to avoid hardware memory limitation problem in mesh generation of billion order DOF model. Load imbalance is addressed by developing a load balancing procedure based on two-dimensional irregular grid, and by managing the workload of threads in the implementation of hybrid MPI-OpenMP. For a simple (flat surface) model with 1.82 billion DOF, the improvement in parallel efficiency reached 27%, compared with the simple hybrid implementation. For a realistic crust model with about 5.8 billion DOF, the improved code resulted to speed up of up to 6.8 in using 8 cores per compute node, when 2,304 nodes of K computer were used.