A Slot-based Virtual Node Method of Consistent Hashing for Optimization of Index Reconfiguration on Distributed Search

抄録

In the distributed search system, the method of mapping and managing documents on segmented indexes is significant to realize load balancing of distributed search process and efficient cluster reconfiguration. Consistent hashing is the advanced method of data mapping which minimizes the network traffic and redundant index data processing in the index reconfiguration. However, if the cluster consists of several thousands of nodes, it requires huge memory resources. Furthermore, it takes a long time to execute the index reconfiguration because of the overhead of many index splitting processes. In this paper, we propose a new method called slot-based virtual node method of consistent hashing to solve the above issues. As the multiple nodes are added or removed, our new method plots or reallocates the virtual nodes on the hash ring space to realize “a bunch of” data migration as far as possible to optimize the index reconfiguration. Slot-based virtual node management saves memory consumption for the mapping information. We actually evaluate memory consumptions of both conventional and our proposed methods to bring out the resource-saving effect. Besides, we estimate lapse times of index reconfiguration processes based on the data processing models to verify the effective reduction of time in our method.