Cross-Era Evaluation of Language Models for Location Referring Expression Extraction

Abstract

Automatic extraction of location referring expressions (LREs) can facilitate humanities research by enabling the analysis of large collections of historical texts. In this study, we constructed LRE annotation datasets from early modern and modern travelogues. We then evaluated the performance of Transformer-based contemporary language models in extracting LREs from historical texts by combining these datasets with existing datasets of modern disaster records and contemporary travelogues. Our experiments demonstrated the effectiveness of leveraging contemporary annotated data for LRE extraction from historical texts. However, whereas extraction accuracy on contemporary texts was high (maximum F1 score of 0.890), accuracy on historical texts remained low to moderate (maximum F1 scores of 0.506–0.739), indicating that further model enhancements are needed to better adapt contemporary language models to historical text.