Analyzing and Evaluating the Performance of a Case-Based Reasoning System

Abstract

A case-based reasoning (CBR) system solves new problems by adapting relevant cases from a case library (a case-base). When we use CBR for practical applications, we usually meet several trade-offs between CBR systems and conventional ones. In practice, the complex integration of case-based reasoners and conventional problem solvers may cause bad performance of both problem solving processes and learning processes. In this paper, we give critical analyses on the nature of CBR as a speed-up and memory-based learner, and propose quantitative measures to evaluate the performance of a CBR application system. The measures are Space Partition Ratio (SPR), Analogical Adaptation Ratio (AAF), and Reusable Case Ratio (RCR). SPR shows the utility of indices of stored cases. AAF shows the performance of adaptation of stored cases to a new problem. RCR shows direct usability of stored cases to a new problem. To validate the effectiveness of the measures, we have carried out intensive experiments on a system : IRS-CBR (Intelligent Information Retriever with a Case-Based Reasoner), which adapts the CBR method to the task of information retrieval for financial statistical databases. From the experimental results, we conclude that (1) IRS-CBR has the advantage for improving the performance of conventional information retrieval systems, (2) IRS-CBR has succeeded in both speed-up and memory-based learning in a practical sense, and (3) the proposed measures are useful for the evaluation of a CBR application.