Frame Popularity-Aware Loss-Resilient Interactive Multi-View Video Streaming

Abstract

Multi-view video streaming plays an important role in new interactive and augmented video applications such as telepresence, remote surgery, and entertainment. For those applications, interactive multi-view video transmission schemes have been proposed that aim to reduce the amount of video traffic. Specifically, these schemes only encode and transmit video frames, which are potentially displayed by users, based on periodical feedback from the users. However, existing schemes are vulnerable to frame loss, which often occurs during transmissions, because they encode most video frames using inter prediction and inter-view prediction to reduce traffic. Frame losses induce significant quality degradation due to the collapse of the decoding operations. To improve the loss resilience, we propose an encoding/decoding system, Frame Popularity-based Multi-view Video Streaming (FP-MVS), for interactive multi-view video streaming services. The main idea of FP-MVS is to assign intra (I) frames in the prediction structure for less/more popular (i.e., few/many observed users) potential frames in order to mitigate the impact of a frame loss. In addition, FP-MVS utilizes overlapping and non-overlapping areas between all user's potential frames to prevent redundant video transmission. Although each intra-frame has a large data size, the video traffic can be reduced within a network constraint by combining multicast and unicast for overlapping and non-overlapping area transmissions. Evaluations using Joint Multi-view Video Coding (JMVC) demonstrated that FP-MVS achieves higher video quality even in loss-prone environments. For example, our scheme improves video quality by 11.81dB compared to the standard multi-view video encoding schemes at the loss rate of 5%.