Question about using full-item evaluation ("all") instead of negative sampling

[tomatoboy-hub]

Hi, @Yuhanleeee

Thanks for sharing this great repository! I'm currently researching recommendation systems and have a quick question about your evaluation setup.

I noticed that you use full-item evaluation (evaluating on "all" unobserved items) rather than negative sampling. I'm curious about the reasoning behind this choice, especially since I often see other papers and repos using 100 or 1000 negative samples to reduce computational cost.

Is it primarily to avoid the ranking inconsistency issues (like the ones pointed out in KDD 2020), or are there other specific reasons for this choice in your project?

I'd love to hear your thoughts if you have a moment. Thanks!

[Yuhanleeee]

Thanks for your attention and kind question. We adopt full-item evaluation primarily because it provides a precise and unbiased estimate of true ranking performance, whereas negative sampling introduces bias and variance into metrics, as highlighted in “On Sampled Metrics for Item Recommendation.” In particular, sampled metrics can distort both absolute scores and relative model rankings due to the altered candidate distribution, leading to potential ranking inconsistency across evaluation protocols. By evaluating all items, we eliminate dependence on sampling strategies, ensuring faithful, reproducible, and directly comparable results that better reflect real-world deployment scenarios.