It is common practice to pre-orthogonalise data prior to training diagonal covariance continuous mixture density Hidden Markov Models (HMMs); this data orthogonalisation process is typically seen as being separate and distinct from the HMM training itself. Thus training is a two stage process, first the data is orthogonalised via a linear transformation, and then the HMM parameters are re-estimated using this pre-orthogonalised data. In this paper we provide an alternative interpretation; we show that observation vector orthogonalisation can be interpreted as simply another form of HMM mixture tying. Hence an HMM trained using orthogonalised data is identically equivalent to a form of tied HMM. Extending the basic orthogonalisation scheme described above we introduce a model dependent observation vector orthogonalisation algorithm, where each HMM is associated with its own separate model dependent orthogonalisation matrix. Using the theoretical framework described above, the corresponding tied mixture interpretation is derived. Finally, experimental results are presented comparing all of the various approaches on a common Japanese phoneme recognition task.