Flow-Join++: Skew-aware Distributed Joins with Cardinality-based Heavy Hitter Prediction

Abstract

In distributed data processing for OLAP workloads, data skew is a critical issue when joins are involved. This skew causes a load imbalance as heavy hitters create straggler nodes and increase execution times. One method that addresses this issue is Flow-Join, which prevents heavy hitters from being transferred and processed on a single node. However, it has two limitations: time-consuming heavy hitter detection and a lack of plan-level coordination in multi-table join workloads. In this paper, we propose Flow-Join++, a skew-aware distributed join approach that leverages a cardinality estimation model to predict heavy hitters and optimize query execution at the plan level. Flow-Join++ estimates the fanout of join keys in a single inference step, eliminating sampling overhead for heavy hitter detection. It provides a Heavy Hitter Push-Down strategy that partitions heavy hitter tuples immediately after scanning base relations, enabling coordinated skew-aware processing across multiple joins. We conducted experiments using the TPC-H benchmark with controllable skew injections, as well as the Phone Bill benchmark, which simulates realistic OLAP workloads in telecommunications services. The results validate the effectiveness of the proposed approach compared to standard join strategies.