As an extension of the immersed boundary method with unified interpolation stencil (Kor, Badri Ghomizad, and Fukagata, J. Fluid Sci. Technol., Vol. 12, 2017, JFST0011), we propose an immersed boundary method that can handle moving boundary problems with a lower level of spurious force oscillation. The key modification to the previously proposed method, which was validated for fixed boundary problems, is to adopt the reconstruction method, in which the velocities outside the body are reconstructed, instead of the ghost-cell method, in which the velocities inside the body are set to satisfy the boundary conditions. From the comparison between the ghost-cell and reconstruction methods, both methods work equally well for a fixed boundary problem, but the reconstruction method is found to be effective in suppressing the spurious force oscillations that appear in moving boundary problems. The capability of the proposed method is demonstrated by numerical investigations of some typical problems. Both predefined motions, such an oscillating cylinder and a hovering flat plate, and interacting motions of rigid bodies are simulated to validate the method. For the latter, sedimentation of a single cylinder as well as a group of interacting cylinders under the gravitational force is examined to demonstrate the capability of the present method for fluid-structure interaction problems. The results show that the proposed method can properly handle the moving boundary problems, while preserving the simplicity of the unified interpolation stencil.