We propose a kernel-based quadratic classification method based on kernel principal component analysis (KPCA). Subspace methods have been widely used for multiclass classification problems, and they have been extended by the kernel trick. However, there are large computational complexities for the subspace methods that use the kernel trick because the problems are defined in the space spanned by all of the training samples. To reduce the computational complexity of the subspace methods for multiclass classification problems, we extend Oja's averaged learning subspace method and apply a subset approximation of KPCA. We also propose an efficient method for selecting the basis vectors for this. Due to these extensions, for many problems, our classification method exhibits a higher classification accuracy with fewer basis vectors than does the support vector machine (SVM) or conventional subspace methods.