A General Scheme for Solving Device Heterogeneity in Consistent Hash Algorithms

抄録

Large-scale interactive systems require efficient performance and scalability to accommodate user demands. The response time of individual requests can be a limiting factor for overall performance. One effective solution is to employ a consistent hashing algorithm for request distribution, which alleviates the load on each server and enhances response speed. However, existing consistent hash algorithms often fail to account for the heterogeneity of resources, which may degrade performance and result in underutilized resources. To solve this problem, this paper proposes a solution based on the study of cluster load balancing using a consistent hash algorithm to increase the resource utilization. Firstly, The scheme adopts a two-dimensional spatial coordinate positioning. Partitions are formed by dividing the ring created by the value range of the CRC32 or CRC64 hash function. The partition and relative position for a key are determined by its vertical and horizontal coordinate, respectively. Secondly, nodes with different performance are assigned with different weights, and each partition also has a corresponding weight which determined by the number and weight of nodes within it. Experiments have shown that this scheme can achieve a search rate of more than 30 million keys per second, with a resource utilization rate of over 80% within the cluster, and maintain stability when changing the number of nodes, thereby improving the overall performance of the system.