IEICE Transactions on Communications
Online ISSN : 1745-1345
Print ISSN : 0916-8516
Special Section on Management for the Era of Internet of Things and Big Data
Managing the Synchronization in the Lambda Architecture for Optimized Big Data Analysis
Thomas VANHOVEGregory VAN SEGHBROECKTim WAUTERSBruno VOLCKAERTFilip DE TURCK
Author information
JOURNAL FREE ACCESS

2016 Volume E99.B Issue 2 Pages 297-306

Details
Abstract

In a world of continuously expanding amounts of data, retrieving interesting information from enormous data sets becomes more complex every day. Solutions for precomputing views on these big data sets mostly follow either an offline approach, which is slow but can take into account the entire data set, or a streaming approach, which is fast but only relies on the latest data entries. A hybrid solution was introduced through the Lambda architecture concept. It combines both offline and streaming approaches by analyzing data in a fast speed layer first, and in a slower batch layer later. However, this introduces a new synchronization challenge: once the data is analyzed by the batch layer, the corresponding information needs to be removed in the speed layer without introducing redundancy or loss of data. In this paper we propose a new approach to implement the Lambda architecture concept independent of the technologies used for offline and stream computing. A universal solution is provided to manage the complex synchronization introduced by the Lambda architecture and techniques to provide fault tolerance. The proposed solution is evaluated by means of detailed experimental results.

Content from these authors
© 2016 The Institute of Electronics, Information and Communication Engineers
Previous article Next article
feedback
Top