Detecting small infrared targets is a difficult but important task in highly cluttered coastal surveillance. The paper proposed a method called low-rank and sparse decomposition based frame difference to improve the detection performance of a surveillance system. First, the frame difference is used in adjacent frames to detect the candidate object regions which we are most interested in. Then we further exclude clutters by low-rank and sparse matrix recovery. Finally, the targets are extracted from the recovered target component by a local self-adaptive threshold. The experiment results show that, the method could effectively enhance the system's signal-to-clutter ratio gain and background suppression factor, and precisely extract target in highly cluttered coastal scene.