Reproducibility of experiments in recommender systems evaluation

Polatidis, Nikolaos; Kapetanakis, Stelios; Pimenidis, Elias; Kosmidis, Konstantinos

Abstract

© IFIP International Federation for Information Processing 2018 Published by Springer International Publishing AG 2018. All Rights Reserved. Recommender systems evaluation is usually based on predictive accuracy metrics with better scores meaning recommendations of higher quality. However, the comparison of results is becoming increasingly difficult, since there are different recommendation frameworks and different settings in the design and implementation of the experiments. Furthermore, there might be minor differences on algorithm implementation among the different frameworks. In this paper, we compare well known recommendation algorithms, using the same dataset, metrics and overall settings, the results of which point to result differences across frameworks with the exact same settings. Hence, we propose the use of standards that should be followed as guidelines to ensure the replication of experiments and the reproducibility of the results.

• Presentation Conference Type: Conference Paper (published)
• Conference Name: 14th International Conference on Artificial Intelligence Applications and Innovations
• Acceptance Date: Apr 11, 2018
• Online Publication Date: May 22, 2018
• Publication Date: Jan 1, 2018
• Deposit Date: May 16, 2018
• Publicly Available Date: May 16, 2018
• Journal: IFIP Advances in Information and Communication Technology
• Print ISSN: 1868-4238
• Electronic ISSN: 1868-422X
• Publisher: Springer Verlag (Germany)
• Peer Reviewed: Peer Reviewed
• Volume: 519
• Pages: 401-409
• Series Title: IFIP Advances in Information and Communication Technology
• DOI: https://doi.org/10.1007/978-3-319-92007-8_34
• Keywords: recommender systems, evaluation, reproducibility, replication
• Public URL: https://uwe-repository.worktribe.com/output/869675
• Publisher URL: https://doi.org/10.1007/978-3-319-92007-8_34
• Contract Date: May 16, 2018