Investigation of ChatGPT's Capabilities for Modeling Education

Abstract

In recent years, large language models (LLMs) like ChatGPT have been used in many fields. Some studies have tested LLMs' ability to create UML models and solve difficult exams. However, no study has tested LLMs on UML modeling exams. This study examines how well LLMs can answer questions from the UMTP UML Modeling Certification Exam (UMTP), especially questions about class diagrams. We also explore how ChatGPT can help with learning. We tested ChatGPT-o1 and ChatGPT-4o on UMTP class diagram questions. The results showed that ChatGPT-o1 answered more questions correctly than ChatGPT-4o. We also found that ChatGPT-o1 explained its reasoning step by step. This step-by-step reasoning may not only help in finding correct answers but also provide useful explanations for learning. Based on these results, this paper discusses how to use ChatGPT's reasoning process for learning support and what future research is needed.