Development of a dynamic resource allocation system for ML pipelines

Abstract

In a society where various machine learning models are used in practical applications, the importance of managing the machine learning pipelines (ML pipelines), which is a series of processes from data processing to training and inference, is increasing. A ML pipeline platform provides an environment to manage ML pipelines and these computer resources, such as CPU, memory, and GPU. However, current platforms require pre-configuration of computer resources to be used by ML pipelines, and cannot take into account the actual resource usage, resulting in the allocation of extra resources. In this paper, we propose a system that dynamically predicts the amount of computer resources used in each phase of the ML pipeline and executes the ML pipeline with the minimum necessary resources, and evaluate the efficiency of resource use for transfer learning.