DLSA: dual-learning based on self-attention for rating prediction | International Journal of Machine Learning and Cybernetics Skip to main content
Springer Nature Link
Log in

DLSA: dual-learning based on self-attention for rating prediction

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Latent factor models (LFMs) have been widely applied in many rating recommendation systems because of their prediction rating capability. Nevertheless, LFMs may not fully leverage rating information and lack good recommendation performance. Furthermore, many subsequent works have often used auxiliary text information, such as user attributes, to improve the prediction effect. However, they did not fully utilize implicit information (i.e., users’ preferences, items’ common features), and additional information is sometimes difficult to acquire. In this paper, we propose a new framework, named dual-learning based on self-attention for rating prediction (DLSA), to solve these problems. Self-attention has a proven ability to learn implicit information about sentences in machine translation, which can be used to mine implicit information in recommendation systems. Additionally, dual learning has shown that the model can generate feedback information when it learns from unlabeled data; therefore, we were inspired to use it in recommendation and obtain implicit information feedback. From the user’s perspective, we design a user self-attention model to learn user-user implicit information and create an interactive user-item self-attention mechanism to learn user-item information. We can also obtain item self-attention to utilize item-item information and an item-user self-attention model to acquire item-user information from an item’s perspective. The interactive structure of the user-item and item-user can adopt the dual learning mechanism to learn implicit information feedback. Moreover, no auxiliary text information was used in the process. The proposed model combines the power of self-attention for implicit information and dual learning for information feedback in a new neural network architecture. Experiments on several real-world datasets demonstrate the effectiveness of DLSA over competitive algorithms on rating recommendation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. https://www.tensorflow.org.

  2. https://grouplens.org/datasets/movielens/.

References

  1. Adomavicius G, Kwon YO (2011) New recommendation techniques for multicriteria rating systems. IEEE Intell Syst 22(3):48–55

    Article  Google Scholar 

  2. Sarwar BM, Karypis G, Konstan JA, Riedl J (2001) Item-based collaborative filtering recommendation algorithms. In: Proceedings of the tenth international world wide web conference, pp 285–295

  3. Koren Y, Bell R, Volinsky C (2009) Matrix factorization techniques for recommender systems. Computer 42(8):30–37

    Article  Google Scholar 

  4. Wu C, Wu H, Luo C, Wu Q, Liu C, Wu Y, Yang F (2019) Recommendation algorithm based on user score probability and project type. Eur J Wirel Comm 2019:80

  5. Koren Y (2008) Factorization meets the neighborhood: a multifaceted collaborative filtering model. In: Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp 426–434

  6. Lecun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436

    Article  Google Scholar 

  7. Liu X, Ouyang Y, Rong W, Zhang X (2015) Item category aware conditional restricted Boltzmann machine based recommendation. In: International conference on neural information processing, pp 609–616

  8. Zhang L, Luo T, Zhang F, Wu Y (2018) A recommendation model based on deep neural network. IEEE Access 6:9454–9463

    Article  Google Scholar 

  9. Xin D, Lei Y, Wu Z, Sun Y, Yuan L, Zhang F (2017) A hybrid collaborative filtering model with deep structure for recommender systems. In: Proceedings of the thirty-first AAAI conference on artificial intelligence, pp 1309–1315

  10. Vincent P, Larochelle H, Lajoie I, Bengio Y, Manzagol PA (2010) Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J Mach Learn Res 11(12):3371–3408

    MathSciNet  MATH  Google Scholar 

  11. Liu T, Wang Z, Tang J, Yang S, Huang GY, Liu Z (2019) Recommender systems with heterogeneous side information. In: The world wide web conference, pp 3027–3033

  12. Zhao J, Geng X, Zhou J, Sun Q, Xiao Y, Fu Z (2019) Attribute mapping and autoencoder neural network based matrix factorization initialization for recommendation systems. Knowl Based Syst 166:132–139

  13. Yan W, Wang D, Cao M, Liu J (2019) Deep auto encoder model with convolutional text networks for video recommendation. IEEE Access 7:40333–40346

    Article  Google Scholar 

  14. Bi JW, Liu Y, Fan ZP (2019) A deep neural networks based recommendation algorithm using user and item basic data. Int J Mach Learn Cybern 11(4):763–777

  15. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: 2017 annual conference on neural information processing systems, pp 5998–6008

  16. Xia Y, He D, Qin T, et al(2016) Dual learning for machine translation. In: 2016 annual conference on neural information processing systems, pp 820–828

  17. Mnih A, Salakhutdinov RR (2008) Probabilistic matrix factorization. In: Proceedings of the twenty-first annual conference on neural information processing systems, pp 1257–1264

  18. Salakhutdinov R, Mnih A (2008) Bayesian probabilistic matrix factorization using Markov chain Monte Carlo. In: 2008 ICML proceedings of the twenty-fifth international conference, pp 880–887

  19. Lee DD, Seung HS (2001) Algorithms for nonnegative matrix factorization. In: 2000 NIPS papers from neural information processing systems, pp 556–562

  20. Corso D, Gianna M, Romani F (2019) Adaptive nonnegative matrix factorization and measure comparisons for recommender systems. Appl Math Comput 354:164–179

    MathSciNet  MATH  Google Scholar 

  21. Zhao X, Zhu Z, Zhang Y, James C (2020) Improving the estimation of tail ratings in recommender system with multi-latent representations. In: Thirteenth ACM international conference on web search and data mining, 2020 WSDM, Houston, TX, USA, pp 762–770

  22. Rendle S (2010) Factorization machines. In: the 10th IEEE international conference on data mining, pp 995–1000

  23. Lian D, Liu R, Ge Y, Zheng K, Xie X, Cao L (2017) Discrete content-aware matrix factorization. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp 325–344

  24. Wang H, Wang N, Yeung D (2015) Collaborative deep learning for recommender systems. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Sydney, NSW, Australia, 10–13 August 2015, pp 1235–1244

  25. Deng X, Zhuang F, Zhu Z (2019) Neural variational collaborative fltering with side information for top-K recommendation. Int J Mach Learn Cybern 10:3273–3284

    Article  Google Scholar 

  26. Mongia A, Jhamb N, Chouzenoux E, Majumdar A (2020) Deep latent factor model for collaborative filtering. Signal Process 169:107366

  27. Liang H, Baldwin T (2015) A probabilistic rating auto-encoder for personalized recommender systems. In: 2015 CIKM the 24th ACM international conference on information and knowledge management, pp 1863–1866

  28. Zhuang F, Zhang Z, Qian M, Shi C, Xie X, He Q (2017) Representation learning via dualautoencoder for recommendation. Neural Netw 90:83–89

    Article  Google Scholar 

  29. Zhang M, Jiang S, Cui Z, Garnett R, Chen Y (2019) D-vae: A variational autoencoder for directed acyclic graphs. In: 2019 NeurlPS annual conference on neural information processing systems, pp 1586–1598

  30. Scarselli F, Gori M, Tsoi AC, Hagenbuchner M, Monfardini G (2009) The graph neural network model. IEEE T Neural Networ 20(1):61–80

    Article  Google Scholar 

  31. Qian F, Li J, Zhao S, Zhang Y (2019) Rating recommendation based on deep hybrid model. J Nanjing Univ Aeronaut Astronaut 51(5):592–598

    Google Scholar 

  32. Cheng H, Koc L, Harmsen J (2016) Wide & deep learning for recommender systems. In: Proceedings of the 1st workshop on deep learning for recommender systems, pp 7–10

  33. Chen J, Zhang H, He X, Nie L, Liu W, Chua TS (2017) Attentive collaborative filtering: Multimedia recommendation with item-and component-level attention. In: Proceedings of the 40th international acm sigir conference on research and development in information retrieval, pp 335–344

  34. Chen J, Zhuang F, Hong X, Ao X, Xie X, He Q (2018) Attention-driven factor model for explainable personalized recommendation. In: 2018 SIGIR 41st international ACM SIGIR conference on research & development in information retrieval, pp 909–912

  35. Yuan W, Wang H, Yu X, Liu N, Li Z (2020) Attention-based context-aware sequential recommendation model. Inf Sci 510:122–134

    Article  Google Scholar 

  36. Chen J, Wang X, Zhao S, Qian F, Zhang Y (2020) Deep attention user-based collaborative filtering for recommendation. Neurocomputing 383:57–68

    Article  Google Scholar 

  37. Peng D, Yuan W, Liu C (2019) HARSAM: A hybrid model for recommendation supported by self-attention mechanism. IEEE Access 7:12620–12629

    Article  Google Scholar 

  38. Lv Y, Zheng Y, Wei F, Wang C, Wang C (2020) AICF: Attention-based item collaborative filtering. Adv Eng Inform 44:101090

    Article  Google Scholar 

  39. Pang G, Wang X, Hao F, Xie J, Wang X, Lin Y, Qin X (2019) ACNN-FM: A novel recommendation with attention-based convolutional neural network and factorization machines. Knowl Based Syst 181:104786

    Article  Google Scholar 

  40. Xue H, Dai X, Zhang J, Huang S, Chen J (2017) Deep matrix factorization models for recommender systems. In: Proceedings of the 26th international joint conference on artificial intelligence, IJCAI 2017, Melbourne, Australia, 19–25 August 2017, pp 3203–3209

  41. Mikolov T, Sutskever I, Chen K, Corrado G, Dean J (2013) Distributed representations of words and phrases and their compositionality. In: 27th annual conference on neural information processing systems 2013, 5–8 Dec 2013, pp 3111–3119

  42. Jia Y, Wang X, Zhang J (2019) Collaborative filtering via learning characteristics of neighborhood based on convolutional neural networks. In: Proceedings of the 1st international workshop on deep learning practice for high-dimensional sparse data, pp 1–4

  43. Nguyen D, Tsiligianni E, Deligiannis N (2018) Extendable neural matrix completion. In: 2018 IEEE international conference on acoustics, speech and signal processing, ICASSP 2018, Canada, 15–20 April 2018, pp 6328–6332

  44. Li M, Dai J, Zhu F, Zang L, Hu S, Han J (2019) A fuzzy set based approach for rating bias. In: The thirty-third AAAI conference on artificial intelligence, pp 9969–9970

  45. Ma H, Zhou D, Liu C, Lyu MR, King I (2011) Recommender systems with social regularization. In: Proceedings of the Forth international conference on web search and web data mining, 2011 WSDM, Hong Kong, China, pp 287–296

  46. Sabetsarvestani Z, Kiraly F, Miguel R, Rodrigues D (2018) Entry-wise matrix completion from noisy entries. In: 2018 EUSIPCO 26th European signal processing conference, pp 2603–2607

  47. Monti F, Bronstein M, Bresson X (2017) Geometric matrix completion with recurrent multi-graph neural networks. In: Annual conference on neural information processing systems 2017, 4–9 Dec 2017, Long Beach, CA, USA, pp 3697–3707

  48. Liu Y, Zhao P, Liu X, Wu M, Duan L, Li XL (2017) Learning user dependencies for recommendation. In: Proceedings of the twenty-sixth international joint conference on artificial intelligence, IJCAI 2017, 19–25, pp 2379–2385

  49. Zhang M, Chen Y (2019) Inductive graph pattern learning for recommender systems based on a graph neural network. arXiv:1904.12058

  50. Locatello F, Yurtsever A, Fercoq O, Cevher V (2019) Stochastic Frank-Wolfe for composite convex minimization. In: Annual conference on neural information processing systems 2019, NeurIPS 2019, 8–14 December 2019, Vancouver, BC, Canada, pp 14246–14256

  51. Li P, Wang Z, Ren Z, Bing L, Lam W (2017) Neural rating regression with abstractive tips generation for recommendation. In: Proceedings of the 40th international acm sigir conference on research & development in information retrieval, Shinjuku, Tokyo, Japan, 7–11 August 2017, pp 345–354

  52. Bai P, Ge Y, Liu F, Lu H (2019) Joint interaction with context operation for collaborative filtering. Pattern Recogn 88:729–738

    Article  Google Scholar 

Download references

Acknowledgements

This work was jointly supported by the National Key Research and Development Program of China (2017YFB1401903), the Natural Science Foundation of China (Nos. 61673020, 61702003, 61876001), and the Natural Science Foundation of Anhui Province (1808085M-F175). The authors would also like to thank the anonymous reviewers for their valuable comments.

Author information

Author notes

  1. Fulan Qian and Jianhong Li have contributed equally.

Authors and Affiliations

  1. School of Computer Science and Technology, Anhui University, Hefei, China

    Fulan Qian, Yafan Huang, Jianhong Li, Shu Zhao & Yanping Zhang

  2. Key Laboratory of Intelligent Computing and Signal Processing, Ministry of Education, Hefei, China

    Fulan Qian, Yafan Huang, Jianhong Li, Shu Zhao & Yanping Zhang

  3. School of Artificial Intelligence, Anhui University of Science and Technology, Huainan, China

    Jianhong Li & Chengjun Wang

Authors
  1. Fulan Qian
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yafan Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jianhong Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chengjun Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shu Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yanping Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fulan Qian.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qian, F., Huang, Y., Li, J. et al. DLSA: dual-learning based on self-attention for rating prediction. Int. J. Mach. Learn. & Cyber. 12, 1993–2005 (2021). https://doi.org/10.1007/s13042-021-01288-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-021-01288-7

Keywords

Search

Navigation