抄録
This paper examines the problem of evaluating systems that aim at finding one highly relevant document with high precision. Such a task is important for modern search environments such as the Web where recall is unimportant and/or unmeasurable. Reciprocal Rank is a metric designed for finding one relevant document with high precision, but it can only handle binary relevance. We therefore introduce a new metric called O-measure, for high precision, high relevance search, and show (a) How the task of finding one relevant document is different from that of finding as many relevant documents as possible, in both binary and graded relevance settings;and (b) How the task of finding one highly relevant document is different from that of finding any one relevant document. We use four test collections and the corresponding sets of formal runs from the NTCIR-3 Crosslingual Information Retrieval track to compare the tasks and metrics in terms of resemblance, stability and discrimination power in system ranking.
