Abstract
When learning a nonlinear model, we suffer from two difficulties in practice: (1) the local optima, and (2) appropriate model complexity determination problems. As for (1), I recently proposed the split and merge Expectation Maximization (SMEM) algorithm within the framework of the maximum likelihood by simulataneously spliting and merging model components, but the model complexity was fixed there. To overcome these problems, I first formally derive an objective function that can optimize a model over parameter and structure distributions simultaneously based on the variational Bayesian approach. Then, I device a Bayesian SMEM algorithm to e.ciently optimize the objective function. With the proposed algorithm, we can find the optimal model structure while avoiding being trapped in poor local maxima. I apply the proposed method to the learning of a mixture of experts model and show the usefulness of the method.
