IEICE Transactions on Information and Systems
Online ISSN : 1745-1361
Print ISSN : 0916-8532
Regular Section
NDCouplingHDFS: A Coupling Architecture for a Power-Proportional Hadoop Distributed File System
Hieu Hanh LESatoshi HIKIDAHaruo YOKOTA
Author information
JOURNALS FREE ACCESS

Volume E97.D (2014) Issue 2 Pages 213-222

Details
Download PDF (1538K) Contact us
Abstract

Energy-aware distributed file systems are increasingly moving toward power-proportional designs. However, current works have not considered the cost of updating data sets that were modified in a low-power mode, where a subset of nodes were powered off. In detail, when the system moves to a high-power mode, it must internally replicate the updated data to the reactivated nodes. Effectively reflecting the updated data is vital in making a distributed file system, such as the Hadoop Distributed File System (HDFS), power proportional. In the current HDFS design, when the system changes power mode, the block replication process is ineffectively restrained by a single NameNode because of access congestion of the metadata information of blocks. This paper presents a novel architecture, a NameNode and DataNode Coupling Hadoop Distributed File System (NDCouplingHDFS), which effectively reflects the updated blocks when the system goes into high-power mode. This is achieved by coupling metadata management and data management at each node to efficiently localize the range of blocks maintained by the metadata. Experiments using actual machines show that NDCouplingHDFS is able to significantly reduce the execution time required to move updated blocks by 46% relative to the normal HDFS. Moreover, NDCouplingHDFS is capable of increasing the throughput of the system supporting MapReduce by applying an index in metadata management.

Information related to the author
© 2014 The Institute of Electronics, Information and Communication Engineers
Previous article Next article

Recently visited articles
Journal news & Announcements
  • Please contact trans-d [a] ieice.org, if you want to unlock PDF security.
feedback
Top