We propose a novel method of extracting a human silhouette from a golf swing movie taken by a mobile phone camera for automatic golf swing diagnosis. The difficulties of the task are that we cannot use the background subtraction method and that the camera is not fixed. First we correct small camera movements during a swing sequence by tracking features in the obvious background. Based on the fact that the histogram of the pixel value (hue and brightness) at the background pixel distributes in a narrow interval, we classify each pixel into three categories: background pixel, human pixel, and mixture pixel. We propagate pixel values of the background from the background pixels to the mixture ones. Finally we apply a graph cut method to extract the human silhouette at each image. Experimental results showed that the proposed method is promising.