Multi-Task Prediction Method for User Behavior Utilizing Transformers

Abstract

Pattern recognition of user behavior plays an important role in extracting portrait, and critical event sequence extraction is also valuable. Addressing these two issues, an approach of Transformer-based multi-task user behavior prediction is investigated in this paper, named LogSeqTrans model, to enhance the accuracy of predicting user actions and extract critical event sequences. By serializing user behavior data and employing information entropy to identify key events, the proposed LogSeqTrans model processes data through an embedding layer, an encoding layer, and an output layer. The embedding layer converts events and their temporal information into high-dimensional vectors. The encoding layer leverages a multi-head self-attention mechanism to capture sequence dependencies, while the output layer simultaneously predicts behavior types, event occurrence times, and remaining durations. Experimental results demonstrate that the proposed model surpasses other models across three open datasets. Specifically, the average accuracy of LogSeqTrans model for the next activity prediction task significantly outperforming alternative models; Similarly, in the tasks of predicting the next activity occurrence time and the remaining time, the mean absolute errors of LogSeqTrans model are all outperforming comparative models. These results indicate that LogSeqTrans is highly effective in multi-task prediction and capturing complex sequence patterns.