23/08/2020

On sampled metrics for item recommendation

Walid Krichene, Steffen Rendle

Keywords: item recommendation, sampled metric, evaluation, metrics

Abstract Paper Similar Papers

Abstract: The task of item recommendation requires ranking a large catalogue of items given a context. Item recommendation algorithms are evaluated using ranking metrics that depend on the positions of relevant items. To speed up the computation of metrics, recent work often uses sampled metrics where only a smaller set of random items and the relevant items are ranked. This paper investigates sampled metrics in more detail and shows that they are inconsistent with their exact version, in the sense that they do not persist relative statements, e.g., recommender A is better than B, not even in expectation. Moreover, the smaller the sampling size, the less difference there is between metrics, and for very small sampling size, all metrics collapse to the AUC metric. We show that it is possible to improve the quality of the sampled metrics by applying a correction, obtained by minimizing different criteria such as bias or mean squared error. We conclude with an empirical evaluation of the naive sampled metrics and their corrected variants. To summarize, our work suggests that sampling should be avoided for metric calculation, however if an experimental study needs to sample, the proposed corrections can improve the quality of the estimate.

The video of this talk cannot be embedded. You can watch it here:
https://dl.acm.org/doi/10.1145/3394486.3403226#sec-supp
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at KDD 2020 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 15:39 
 0
 0
 0
 0
 3:17 
 0
 0
 0
 0
 15:05 
 0
 0
 0
 0
 18:19 
 0
 0
 0
 0
 10:21 
 0
 0
 0
 0
 12:36 
 0
 0
 0
 0
 18:20 
 0
 0
 0
 0
 5:14 
 0
 0
 0
 0
 4:53 
 0
 0
 0
 0
 13:25 
 0
 0
 0
 0
 12:01 
 0
 0
 0
 0
 3:04 
 0
 0
 0
 0
 3:23 
 0
 0
 0
 0
 12:21 
 0
 0
 0
 0
 15:06 
 0
 0
 0
 0
 10:35 
 0
 0
 0
 0
 7:49 
 0
 0
 0
 0
 15:18 
 0
 0
 0
 0
 14:59 
 0
 0
 0
 0
 11:53 
 0
 0
 0
 0
 13:48 
 0
 0
 0
 0
 5:21 
 0
 0
 0
 0
 15:30 
 0
 0
 0
 0
 3:20 
 0
 0
 0
 0
 4:50 
 0
 0
 0
 0
 8:03 
 0
 0
 0
 0
 5:20 
 0
 0
 0
 0
 17:10 
 0
 0
 0
 0
 3:05 
 0
 0
 0
 0
 6:57 
 0
 0
 0
 0
 3:01 
 0
 0
 0
 0
 1:01 
 0
 0
 0
 0
 15:08 
 0
 0
 0
 0
 17:01 
 0
 0
 0
 0
 15:40 
 0
 0
 0
 0
 5:45 
 0
 0
 0
 0
 5:12 
 0
 0
 0
 0
 14:03 
 0
 0
 0
 0
 3:23 
 0
 0
 0
 0
 18:01