Recommender Systems: Methodological Challenges and Issues
https://doi.org/10.26794/2304-022X-2026-16-2-124-135
Abstract
The aim of this study is to analyse the evolution of methods for building recommender systems and approaches to evaluating their performance under conditions of multi-criteria decision-making and dynamically changing user preferences. The paper examines the key stages in the development of recommender technologies and systematises modern algorithms and architectures, including hybrid, neural network-based, and graph-based models. Particular attention is paid to methods for evaluating recommendation quality and to the classification of metrics reflecting accuracy, ranking quality, diversity, and computational efficiency. The study demonstrates that reliance solely on accuracy metrics is insufficient for an adequate assessment of recommender system performance. The authors address key methodological challenges, including the coldstart problem, data sparsity, and the need to balance accuracy and diversity in recommendations. The paper concludes that a comprehensive multi-criteria approach is essential for the effective evaluation of recommender systems. The findings may be useful for researchers, recommender system developers, and specialists in machine learning and data analysis, as well as for business practitioners involved in selecting and evaluating the effectiveness of recommendation algorithms.
Keywords
recommender systems,
recommendation quality evaluation,
development trends,
personalization,
models,
algorithms,
metrics
Supporting agencies: The work was carried out within the framework of the state assignment of the Ministry of Education and Science of the Russian Federation; project FEWM-2026-0011.
About the Authors
R. S. Kulshin
Tomsk State University of Control Systems and Radioelectronics
Russian Federation
Russian Federation
Roman S. Kulshin – Senior lecturer of the Department of Data Processing Automation, Postgraduate student of the Department of Data Processing Automation
Tomsk
A. A. Sidorov
Tomsk State University of Control Systems and Radioelectronics
Russian Federation
Russian Federation
Anatoly A. Sidorov – Cand. Sci. (Econ.), Assoc. Prof.; Head of the Department of Data Processing Automation
Tomsk
References
1. Goldberg D., Nichols D., Oki B. M., Terry D. Using collaborative filtering to weave an information tapestry. Communications of the ACM. 1992;35(12):61-70. DOI: 10.1145/138859.138867
2. Resnick P., Iacovou N., Suchak M., Bergstrom P., Riedl J. Grouplens: An open architecture for collaborative filtering of netnews. In: Proc. 1994 ACM conf. on computer supported cooperative work. New York, NY: Association for Computing Machinery; 1994:175-186. DOI: 10.1145/192844.192905
3. Schafer J.B., Konstan J., Riedl J. Recommender systems in e-commerce. In: Proc. 1st ACM conf. on electronic commerce. New York, NY: Association for Computing Machinery; 1999:158-166. DOI: 10.1145/336992.337035
4. Bennett J., Elkan C., Liu B., Smyth P., Tikk D. Kdd Cup and workshop 2007. ACM SIGKDD Explorations Newsletter. 2007;9(2):51-52. DOI: 10.1145/1345448.1345459
5. Koren Y., Bell R., Volinsky C. Matrix factorization techniques for recommender systems. Computer. 2009;42(8):30-37. DOI: 10.1109/MC.2009.263
6. Burke R. Hybrid recommender systems: Survey and experiments. User Modeling and User-Adapted Interaction. 2002;12(4):331-370. DOI: 10.1023/A:1021240730564
7. Herlocker J.L., Konstan J.A., Terveen L.G., Riedl J.T. Evaluating collaborative filtering recommender systems. ACM Transactions on Information Systems (TOIS). 2004;22(1):5-53. DOI: 10.1145/963770.963772
8. Linden G., Smith B., York J. Amazon.com recommendations: Item-to-item collaborative filtering. IEEE Internet Computing. 2003;7(1):76-80. DOI: 10.1109/MIC.2003.1167344
9. Lops P., De Gemmis M., Semeraro G. Content-based recommender systems: State of the art and trends. In: Ricci F., Rokach L., Shapira B., Kantor P., eds. Recommender systems handbook. Boston, MA: Springer; 2011:73-105. DOI: 10.1007/978-0-387-85820-3_3
10. Celma Ò. Music recommendation. In: Music recommendation and discovery: The long tail, long fail, and long play in the digital music space. Berlin, Heidelberg: Springer; 2010:43-85. DOI: 10.1007/978-3-642-13287-2_3
11. Lang K. Newsweeder: Learning to filter netnews. In: Prieditis A., Russell S.J., eds. Proc. 12th Int. conf. on machine learning (ICML’95). (Tahoe City, CA, July 9-12, 1995). San Francisco, CA; Morgan Kaufmann Publishers Inc.; 1995:331-339. DOI: 10.1016/B978-1-55860-377-6.50048-7
12. Burke R. Hybrid web recommender systems. In: Brusilovsky P., Kobsa A., Nejdl W., eds. The adaptive web: Methods and strategies of web personalization. Berlin, Heidelberg: Springer; 2007:377-408. (Lecture notes in computer science. Vol. 4321). DOI: 10.1007/978-3-540-72079-9_12
13. Gomez-Uribe C.A., Hunt N. The Netflix recommender system: Algorithms, business value, and innovation. ACM Transactions on Management Information Systems (TMIS). 2015;6(4):1-19. DOI: 10.1145/2843948
14. Kanakia A., Shen Z., Eide D., Wang K. A scalable hybrid research paper recommender system for Microsoft Academic. In: The World Wide Web conf. (WWW’19). New York, NY: Association for Computing Machinery; 2019:2893-2899. DOI: 10.1145/3308558.3313700
15. He X., Liao L., Zhang H., et al. Neural collaborative filtering. In: Proc. 26th Int. conf. on World Wide Web (WWW’17). New York, NY: Association for Computing Machinery; 2017:173-182. DOI: 10.1145/3038912.3052569
16. Sedhain S., Menon A.K., Sanner S., Xie L. AutoRec: Autoencoders meet collaborative filtering. In: Proc. 24th Int. conf. on World Wide Web (WWW’15). New York, NY: Association for Computing Machinery; 2015:111-112. DOI: 10.1145/2740908.2742726
17. Covington P., Adams J., Sargin E. Deep neural networks for YouTube recommendations. In: Proc. 10th ACM conf. on recommender systems (RecSys’16). New York, NY: Association for Computing Machinery; 2016:191-198. DOI: 10.1145/2959100.2959190
18. Hidasi B., Karatzoglou A., Baltrunas L., Tikk D. Session-based recommendations with recurrent neural networks. In: 4th Int. conf. on learning representations (ICLR-2016). (San Juan, May 2–4, 2016). Appleton, WI: ICLR; 2016. URL: https://hidasi.eu/assets/pdf/gru4rec_iclr16.pdf
19. Kang W.-C., McAuley J. Self-attentive sequential recommendation. In: 2018 IEEE int. conf. on data mining (ICDM 2018). (Singapore, November 17–20, 2018). New York, NY: IEEE; 2018:197-206. DOI: 10.1109/ICDM.2018.00035
20. Chen Q., Zhao H., Li W., Huang P., Ou W. Behavior sequence transformer for e-commerce recommendation in Alibaba. In: Proc. 1st Int. workshop on deep learning practice for high-dimensional sparse data (DLP-KDD’19). New York, NY: Association for Computing Machinery; 2019:1-4. DOI: 10.1145/3326937.3341261
21. Ying R., He R., Chen K., et al. Graph convolutional neural networks for web-scale recommender systems. In: Proc. 24th ACM SIGKDD Int. conf. on knowledge discovery & data mining (KDD’18). New York, NY: Association for Computing Machinery; 2018:974-983. DOI: 10.1145/3219819.3219890
22. Wang H., Zhang F., Wang J., et al. Ripplenet: Propagating user preferences on the knowledge graph for recommender systems. In: Proc. 27th ACM Int. conf. on information and knowledge management (CIKM’18). New York, NY: Association for Computing Machinery; 2018:417-426. DOI: 10.1145/3269206.3271739
23. Ricci F., Rokach L., Shapira B., eds. Recommender systems handbook. New York, NY: Springer; 2023. 1060 p. DOI: 10.1007/978-1-0716-2197-4
24. Ge M., Delgado-Battenfeld C., Jannach D. Beyond accuracy: Evaluating recommender systems by coverage and serendipity. In: Proc. 4th ACM conf. on recommender systems (RecSys’10). New York, NY: Association for Computing Machinery; 2010:257-260. DOI: 10.1145/1864708.1864761
25. Abdollahpouri H., Adomavicius G., Burke R., et al. Beyond personalization: Research directions in multistakeholder recommendation. 2019. URL: https://www.researchgate.net/publication/332896126_Beyond_Personalization_Research_Directions_in_Multistakeholder_Recommendation
26. Zhang S., Yao L., Sun A., Tay Y. Deep learning based recommender system: A survey and new perspectives. ACM computing surveys (CSUR). 2019;52(1):1-38. DOI: 10.1145/3285029
27. McNee S.M., Riedl J., Konstan J.A. Being accurate is not enough: How accuracy metrics have hurt recommender systems. In: CHI’06 Extended abstracts on human factors in computing systems (CHI EA’06). New York, NY: Association for Computing Machinery; 2006:1097-1101. DOI: 10.1145/1125451.1125659
28. Amatriain X., Castells P., de Vries A., Posse C., Steck H. Workshop on recommendation utility evaluation: Beyond RMSE — RUE 2012. In: Proc. 6th ACM conf. on recommender systems (RecSys’12). New York, NY: Association for Computing Machinery; 2012:351-352. DOI: 10.1145/2365952.2366042
29. Jannach D., Jugovac M. Measuring the business value of recommender systems. ACM Transactions on Management Information Systems (TMIS). 2019;10(4):1-23. DOI: 10.1145/3370082
30. Pu P., Chen L., Hu R. A user-centric evaluation framework for recommender systems. In: Proc. 5th ACM conf. on recommender systems (RecSys’11). New York, NY: Association for Computing Machinery; 2011:157-164. DOI: 10.1145/2043932.2043962
Review
For citations:
Kulshin R.S., Sidorov A.A. Recommender Systems: Methodological Challenges and Issues. Management Sciences. 2026;16(2):124-135. (In Russ.) https://doi.org/10.26794/2304-022X-2026-16-2-124-135
Views:
47
JATS XML
Email this article 
