Fuzzy Entropy-Based Evaluation Framework for Emotional Understanding in Large Language Models

Abstract

This study proposes a novel fuzzy entropy-based evaluation framework for assessing emotional understanding capabilities of Large Language Models (LLMs) in Japanese literary texts. Unlike conventional discrete classification methods, our approach quantifies emotional ambiguity using fuzzy membership functions (Low, Medium, High) and measures evaluation uncertainty through fuzzy entropy ranging from 0.2 to 1.8. We conducted comprehensive experiments with 36 LLM variants across four distinct personas with systematically controlled temperature parameters (0.1-0.9), collecting 4,227 emotional evaluation data points with 97.8% completeness. The analysis revealed significant diversity in emotional understanding mechanisms across developer groups, with positive emotions showing up to 27.2-point differences between companies (Alibaba: 83.0±7.6 vs. xAI: 55.8±24.3). Language-numerical consistency analysis demonstrated substantial variations from 0.554 (Llama-4-Scout-17B) to 0.085 (Qwen3-235B-A22B-FP8), indicating a 6.5-fold difference in evaluation coherence. Our persona-temperature correlation analysis confirmed significant relationships (r = 0.73, p < 0.001), validating the theoretical foundation of cognitive diversity control in LLM evaluation systems.