Hybrid Transformer and Deep Gaussian Process for Parkinson’s Tremor Classification

Abstract

Parkinson’s disease tremor classification from wearable-sensor time series remains challenging due to small datasets, class imbalance, and signal noise. We propose a hybrid model that fuses a Transformer encoder for contextualized feature extraction with a deep Gaussian process (DGP) for probabilistic classification, further accelerated by variational inference (VGP). On the Tremor Challenge benchmark, our method achieves a Test F1 score of 0.4336 using a seven-layer Transformer trained for 3000 epochs and a spectral-mixture kernel. We find that moderate Transformer depth balances representation power against overfitting, and that kernel choice critically interacts with DGP depth: while increasing DGP layers often degrades performance in limited data regimes, spectral-mixture kernels can harness additional hierarchy to improve accuracy. Extended training benefits deeper Transformer variants, suggesting that parameter- rich models require longer optimization. We analyze limitations including dataset size, class imbalance, computational cost, and the omission of uncertainty quantification. We further identify opportunities for improvement via learnable time embeddings (e.g., Time2Vec) and cross-dataset validation. Our results demonstrate the viability of combining self-attention with Bayesian nonparametrics for robust biomedical time-series classification, offering a promising direction for clinical AI applications.