IEICE Transactions on Communications
Online ISSN : 1745-1345
Print ISSN : 0916-8516
Special Section on Management for the Era of Internet of Things and Big Data
Reusing the Results of Queries in MapReduce Systems by Adopting Shared Storage
Zhanye WANGChuanyi LIUDongsheng WANG
Author information
JOURNAL RESTRICTED ACCESS

2016 Volume E99.B Issue 2 Pages 315-325

Details
Abstract

Over the last few years, Apache MapReduce has become the prevailing framework for large scale data processing. Instead of writing MapReduce programs which are too obscure to express, many developers usually adopt high level query languages, such as Hive or Pig Latin, to finish their complex queries. These languages automatically compile each query into a workflow of MapReduce jobs, so they greatly facilitate the querying and management of large datasets. One option to speed up the execution of workflows is to save the results produced previously and reuse them in the future if needed. In this paper we present SuperRack, which uses shared storage devices to store the results of each workflow and allows a new query to reuse these results in order to avoid redundant computation and hasten execution. We propose several novel techniques to improve the access and storage efficiency of the previous results. We also evaluate SuperRack to verify its feasibility and effectiveness. Experiments show that our solution outperforms Hive significantly under TPC-H benchmark and real life workloads.

Content from these authors
© 2016 The Institute of Electronics, Information and Communication Engineers
Previous article Next article
feedback
Top