Scaling Laws of Model Size for World Models

Abstract

With the development of deep learning, significant performance improvements have been achieved in computer vision and natural language processing. In these advancements, scaling laws that demonstrate exponential changes in model performance with respect to model size, dataset size, and computational resources used for training have played a significant role. These scaling laws have been reported to hold for various tasks, including image classification, image generation, and natural language processing. However, it has not yet been verified whether these scaling laws are effective for tasks that involve long-horizon predictions. In this study, we investigate the validity of scaling laws for world models from the perspective of model size. We conduct experiments that scale the model sizes of two world models in a video prediction task conditioned on actions using the CARLA dataset, and verify that the loss function decreases exponentially and the scaling law holds when including large-scale autoencoder.