IEICE Transactions on Information and Systems
Online ISSN : 1745-1361
Print ISSN : 0916-8532
Regular Section
Skew-Tolerant Key Distribution for Load Balancing in MapReduce
Jihoon SONHyunsik CHOIYon Dohn CHUNG
Author information

2012 Volume E95.D Issue 2 Pages 677-680


MapReduce is a parallel processing framework for large scale data. In the reduce phase, MapReduce employs the hash scheme in order to distribute data sharing the same key across cluster nodes. However, this approach is not robust for the skewed data distribution. In this paper, we propose a skew-tolerant key distribution method for MapReduce. The proposed method assigns keys to cluster nodes balancing their workloads. We implemented our proposed method on Hadoop. Through experiments, we evaluate the performance of the proposed method in comparison with the conventional method.

Content from these authors
© 2012 The Institute of Electronics, Information and Communication Engineers
Previous article Next article