An adaptive microphone system is proposed to address the directivity to a specified speaker and track the speaker's motion. This system is constructed on the basis of the generalized sidelobe canceller, and modified to have robust features for the speaker's motion, by adapting the spatio-temporal characteristics of the microphone array. Experiments to track a target sound source and eliminate ambient noises are performed in our laboratory room without anechoic consideration. The experimental results show that the proposed system can track the moving sound source and the noises are clearly reduced.