In this paper, we propose a method of estimating the velocity of a moving object from a series of images. Assuming that the motion in the image plane is described by an affine transformation, we construct a discrete-time observational equation of the motion based on the dynamic motion imagery modeling technique proposed by Schalkoff. But we can not get the model of motion because of the complexity. So, we derive a adaptive Kalman filter algorithm to identify the model and estimate the velocity. Some estimation results are presented to show that the proposed adaptive Kalman filter algorithm is superior to the correlation method without the model.