Human Activity Recognition Framework Based on Generative Imputation of Missing Modalities

Abstract

Human Activity Recognition (HAR) holds significance in understanding and assisting humans and so is required in related applications like healthcare and security. Especially, HAR using machine learning techniques for sensor data measured with wearable devices has been attracting attention because of its high potential. For noise and fault tolerant HAR, we propose a framework that imputes missing modalities in sensor data and recognize human activities simultaneously. Our framework consists of feature extraction by an autoencoder (AE), activity classification by a multilayer perceptron (MLP), and missing modality generation by a conditional generative adversarial network (CGAN), trained by multitask learning. In the experiment, our framework was applied to the CogAge dataset of which task was the recognition of six state activities using two modalities. The framework that was input with only one of the two modalities performed comparably to MLP and the combination of AE and MLP with both modalities.