A Transformer-based fully trainable point process

Abstract

Using prior physical and mathematical knowledge, an appropriate intensity function should be designed when applying a point process to a real-world problem. A novel Transformer-based partially trainable model has been proposed. This model adaptively extracts a sequence feature from the past event sequence using a self-attention mechanism. However, because the feature vector is the transformed vector of the latest event and the intensity function is modeled in a handmade manner given the feature vector, the approximated intensity function and the predicted next event depend strongly on the latest event. To overcome these problems, a novel Transformer-based fully trainable point process (Transformer-FTPP) is proposed. With this model, multiple trainable vectors are transformed through an encoder-decoder Transformer architecture to extract past sequence-representative and future event candidate vectors. This facilitates the realization of an adaptive and general approximation of the intensity function and a prediction of the next event. The effectiveness of the proposed method was proved experimentally using synthetic and real-world event data.