A smoothing spline is used to propose a novel model for the time-varying quantile of the univariate time series using a state space approach. A correlation is further incorporated between the dependent variable and its one-step-ahead quantile. Using a Bayesian approach, an efficient Markov chain Monte Carlo algorithm is described where we use the multi-move sampler, which generates simultaneously latent time-varying quantiles. Numerical examples are provided to show its high sampling efficiency, in comparison with the simple algorithm that generates one latent quantile at a time given other latent quantiles. Furthermore, using Japanese inflation rate data, an empirical analysis is provided with model comparisons.