This paper proposes an on-line rigid object tracking framework via discriminative object appearance modeling and learning. Strong classifiers are combined with 2D scale-rotation invariant local features to treat tracking as a keypoint matching problem. For on-line boosting, we correspond a Gaussian mixture model (GMM) to each weak classifier and propose a GMM-based classifying mechanism. Meanwhile, self-organizing theory is applied to perform automatic clustering for sequential updating. Benefiting from the invariance of the SURF feature and the proposed on-line classifying technique, we can easily find reliable matching pairs and thus perform accurate and stable tracking. Experiments show that the proposed method achieves better performance than previously reported trackers.