An Efficient and Scalable Distributed Hypergraph Processing System

Abstract

The recent increase in the amount of graph data has drawn our attention to distributed graph processing systems scalable to large-scale inputs. Although distributed-memory processing is generally less efficient than shared-memory processing because of the communication costs and program complexity, state-of-the-art distributed graph processing systems, such as Gemini, have achieved a comparable efficiency by using lightweight graph partitioning and load balancing. However, the achievement of both scalability and efficiency in hypergraph processing remains an open issue because distributed hypergraph processing systems have not been extensively studied. In this paper, we propose a distributed hypergraph processing system based on Gemini that achieves both scalability and efficiency. Our system outperformed the state-of-the-art shared-memory hypergraph processing system Hygra from several folds to tens of folds on a single-node computer. In addition, it showed speedup in processing a large-scale dataset on a multi-node computer.