Information Retrieval (IR) test collections are growing larger, and relevance data constructed through
pooling are suspected of becoming more and more
incomplete and
biased. Several studies have used IR evaluation metrics specifically designed to handle this problem, but most of them have only examined the metrics under
incomplete but unbiased conditions, using random samples of the original relevance data. This paper examines nine metrics in more realistic settings, by reducing the number of pooled systems and the number of pooled documents. Even though previous studies have shown that metrics based on a
condensed list, obtained by removing all unjudged documents from the original ranked list, are effective for handling very incomplete but unbiased relevance data, we show that these results do not hold when the relevance data are biased towards particular systems or towards the top of the pools. More specifically, we show that the condensed-list versions of
Average Precision,
Q-measure and
normalised Discounted Cumulative Gain, which we denote as AP', Q' and nDCG', are not necessarily superior to the original metrics for handling biases. Nevertheless, AP' and Q'
are generally superior to
bpref,
Rank-Biased Precision and its condensed-list version even in the presence of biases.
抄録全体を表示