Structuring and Evaluating Specialized Knowledge through Knowledge Graph Construction

Abstract

In this study, we propose storing specialized, hierarchically structured short texts in a knowledge graph and leveraging large language models (LLMs) for question answering to improve the use of domain-specific knowledge within organizations. While conventional GraphRAG effectively extracts general knowledge from large datasets, it struggles with more limited and specialized texts. To address this, we introduce GraphRAG-hierarchy, which directly converts hierarchically structured documents into a graph, and compare it with both standard GraphRAG and a hybrid approach that combines GraphRAG with vector search (Hybrid-RAG). Our evaluations show that GraphRAG-hierarchy yields higher-precision responses for hierarchical documents, while Hybrid-RAG improves specificity and comprehensiveness but significantly increases the required input context. These findings suggest that our method can enhance knowledge sharing and retrieval in secure, local environments. Future work includes establishing quantitative metrics, further refining response accuracy, and optimizing operational costs, ultimately supporting DX initiatives in highly specialized fields.