Semantic Motion Signature for Segmentation of High Speed Large Displacement Objects

Abstract

This paper presents a novel method for unsupervised segmentation of objects with large displacements in high speed video sequences. Our general framework introduces a new foreground object predicting method that finds object hypotheses by encoding both spatial and temporal features via a semantic motion signature scheme. More specifically, temporal cues of object hypotheses are captured by the motion signature proposed in this paper, which is derived from sparse saliency representation imposed on magnitude of optical flow field. We integrate semantic scores derived from deep networks with location priors that allows us to directly estimate appearance potentials of foreground hypotheses. A unified MRF energy functional is proposed to simultaneously incorporate the information from the motion signature and semantic prediction features. The functional enforces both spatial and temporal consistency and impose appearance constancy and spatio-temporal smoothness constraints directly on the object hypotheses. It inherently handles the challenges of segmenting ambiguous objects with large displacements in high speed videos. Our experiments on video object segmentation benchmarks demonstrate the effectiveness of the proposed method for segmenting high speed objects despite the complicated scene dynamics and large displacements.