Formation of Geospatial Representations in Large Language Models and the Effect of Training Data

Abstract

Large language models (LLMs) have demonstrated the ability to solve tasks in geographic domains, and it has been suggested that these capabilities rely on an internal geospatial world model. However, previous studies have mainly examined such representations using only a small number of the models trained on English-centric data, leaving it unclear how geospatial representations emerge in some models trained on other languages. In this study, we investigate the internal geographic representations of multiple regions in models pre-trained on data in different languages. Our experimental results indicate that the properties of these world models may strongly depend on the language used during training.