{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,1,21]],"date-time":"2025-01-21T05:17:50Z","timestamp":1737436670330,"version":"3.33.0"},"reference-count":45,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2023,2,27]],"date-time":"2023-02-27T00:00:00Z","timestamp":1677456000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100018919","name":"Peng Cheng Laboratory","doi-asserted-by":"crossref","award":["PCL2021A10"],"id":[{"id":"10.13039\/100018919","id-type":"DOI","asserted-by":"crossref"}]},{"name":"the China National Key R&D Program of China","award":["2020YFB1807600"]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"The rapid development of mobile communication services in recent years has resulted in a scarcity of spectrum resources. This paper addresses the problem of multi-dimensional resource allocation in cognitive radio systems. Deep reinforcement learning (DRL) combines deep learning and reinforcement learning to enable agents to solve complex problems. In this study, we propose a training approach based on DRL to design a strategy for secondary users in the communication system to share the spectrum and control their transmission power. The neural networks are constructed using the Deep Q-Network and Deep Recurrent Q-Network structures. The results of the conducted simulation experiments demonstrate that the proposed method can effectively improve the user\u2019s reward and reduce collisions. In terms of reward, the proposed method outperforms opportunistic multichannel ALOHA by about 10% and about 30% for the single SU scenario and the multi-SU scenario, respectively. Furthermore, we explore the complexity of the algorithm and the influence of parameters in the DRL algorithm on the training.<\/jats:p>","DOI":"10.3390\/s23052622","type":"journal-article","created":{"date-parts":[[2023,2,28]],"date-time":"2023-02-28T07:01:51Z","timestamp":1677567711000},"page":"2622","source":"Crossref","is-referenced-by-count":7,"title":["Dynamic Spectrum Sharing Based on Deep Reinforcement Learning in Mobile Communication Systems"],"prefix":"10.3390","volume":"23","author":[{"given":"Sizhuang","family":"Liu","sequence":"first","affiliation":[{"name":"Department of Electronic Engineering, Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing 100084, China"}]},{"given":"Changyong","family":"Pan","sequence":"additional","affiliation":[{"name":"Department of Electronic Engineering, Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing 100084, China"},{"name":"Peng Cheng Laboratory, Shenzhen 518055, China"}]},{"given":"Chao","family":"Zhang","sequence":"additional","affiliation":[{"name":"Department of Electronic Engineering, Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing 100084, China"},{"name":"Peng Cheng Laboratory, Shenzhen 518055, China"}]},{"given":"Fang","family":"Yang","sequence":"additional","affiliation":[{"name":"Department of Electronic Engineering, Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing 100084, China"},{"name":"Peng Cheng Laboratory, Shenzhen 518055, China"}]},{"given":"Jian","family":"Song","sequence":"additional","affiliation":[{"name":"Department of Electronic Engineering, Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing 100084, China"},{"name":"Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China"}]}],"member":"1968","published-online":{"date-parts":[[2023,2,27]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1938","DOI":"10.1109\/JIOT.2018.2872441","article-title":"Distributive dynamic spectrum access through deep reinforcement learning: A reservoir computing-based approach","volume":"6","author":"Chang","year":"2019","journal-title":"IEEE Internet Things J."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Zong, J., Liu, Y., Liu, H., Wang, Q., and Chen, P. (2022, January 27\u201329). 6G Cell-Free Network Architecture. In Proceedings of the 2022 IEEE 2nd International Conference on Electronic Technology, Communication and Information (ICETCI), Changchun, China.","DOI":"10.1109\/ICETCI55101.2022.9832308"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"310","DOI":"10.1109\/TWC.2018.2879433","article-title":"Deep multi-user reinforcement learning for distributed dynamic spectrum access","volume":"18","author":"Naparstek","year":"2013","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_4","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Liu, S., Wang, T., Pan, C., Zhang, C., Yang, F., and Song, J. (2021, January 4\u20136). Deep reinforcement learning for spectrum sharing in future mobile communication system. Proceedings of the 2021 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB), Chengdu, China.","DOI":"10.1109\/BMSB53066.2021.9547161"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1109\/JSAC.2004.839380","article-title":"Cognitive radio: Brain-empowered wireless communications","volume":"23","author":"Haykin","year":"2005","journal-title":"IEEE J. Select. Areas Commun."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Zhou, W., Zhu, Q., and Ling, Y. (2010, January 12\u201314). An improved game-theoretic algorithm for competitive spectrum sharing. Proceedings of the 2010 International Conference on Communications and Mobile Computing, Shenzhen, China.","DOI":"10.1109\/CMC.2010.159"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Cai, F., Gao, Y., Cheng, L., Sang, L., and Yang, D. (2016, January 3\u20136). Spectrum sharing for LTE and WiFi coexistence using decision tree and game theory. Proceedings of the 2016 IEEE Wireless Communications and Networking Conference, Doha, Qatar.","DOI":"10.1109\/WCNC.2016.7565015"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Ahmad, I., Wei, Z., Feng, Z., Bai, Y., Zhang, Q., and Zhang, P. (2014, January 1\u20134). Joint price and power allocation under interference constraint for dynamic spectrum access networks. Proceedings of the 2014 IEEE International Symposium on Dynamic Spectrum Access Networks, McLean, VA, USA.","DOI":"10.1109\/DySPAN.2014.6817790"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Pandit, S., and Singh, G. (2013, January 22\u201323). Spectrum sharing in cognitive radio using game theory. Proceedings of the 2013 3rd IEEE International Advance Computing Conference (IACC), Ghaziabad, India.","DOI":"10.1109\/IAdCC.2013.6514449"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Zhang, N., Yang, D., and Jing, L. (2019, January 11\u201313). An advanced algorithm for spectrum allocation of primary users based on cournot game. Proceedings of the 2019 IEEE International Conference on Signal, Information and Data Processing (ICSIDP), Chongqing, China.","DOI":"10.1109\/ICSIDP47821.2019.9173393"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Alhammadi, A., Roslee, M., and Alias, M.Y. (2016, January 28\u201330). Fuzzy logic based negotiation approach for spectrum handoff in cognitive radio network. Proceedings of the 2016 IEEE 3rd International Symposium on Telecommunication Technologies (ISTT), Kuala Lumpur, Malaysia.","DOI":"10.1109\/ISTT.2016.7918097"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Alhammadi, A., Roslee, M., and Alias, M.Y. (2016, January 28\u201330). Analysis of spectrum handoff schemes in cognitive radio network using particle swarm optimization. Proceedings of the 2016 IEEE 3rd International Symposium on Telecommunication Technologies (ISTT), Kuala Lumpur, Malaysia.","DOI":"10.1109\/ISTT.2016.7918093"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Roslee, M., Alhammadi, A., Alias, M.Y., and Nmenme, P.U. (2017, January 6\u20138). Efficient handoff spectrum scheme using fuzzy decision making in cognitive radio system. Proceedings of the 2017 3rd International Conference on Frontiers of Signal Processing (ICFSP), Paris, France.","DOI":"10.1109\/ICFSP.2017.8097144"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"3133","DOI":"10.1109\/COMST.2019.2916583","article-title":"Applications of deep reinforcement learning in communications and networking: A survey","volume":"21","author":"Luong","year":"2019","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Gao, X., Dou, Z., and Qi, L. (2020, January 6\u20139). A new distributed dynamic spectrum access model based on DQN. Proceedings of the 2020 15th IEEE International Conference on Signal Processing, Beijing, China.","DOI":"10.1109\/ICSP48669.2020.9320903"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Nguyen, H.Q., Nguyen, B.T., Dong, T.Q., Ngo, D.T., and Nguyen, T.A. (2018, January 22\u201325). Deep Q-Learning with multiband sensing for dynamic spectrum access. Proceedings of the 2018 IEEE International Symposium on Dynamic Spectrum Access Networks (DySPAN), Seoul, Republic of Korea.","DOI":"10.1109\/DySPAN.2018.8610402"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1109\/TCCN.2018.2809722","article-title":"Deep reinforcement learning for dynamic multichannel access in wireless networks","volume":"4","author":"Wang","year":"2018","journal-title":"IEEE Trans. Cogn. Commun. Netw."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"25463","DOI":"10.1109\/ACCESS.2018.2831240","article-title":"Intelligent power control for spectrum sharing in cognitive radios: A deep reinforcement learning approach","volume":"6","author":"Li","year":"2018","journal-title":"IEEE Access"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Wang, X., Teraki, Y., Umehira, M., Zhou, H., and Ji, Y. (2022). A usage aware dynamic spectrum access scheme for interweave cognitive radio network by exploiting deep reinforcement learning. Sensors, 22.","DOI":"10.3390\/s22186949"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Ye, F., Zhang, Y., Li, Y., Jiang, T., and Li, Y. (2020, January 5\u201310). Power control based on deep Q network with modified reward function in cognitive networks. Proceedings of the 2020 IEEE USNC-CNC-URSI North American Radio Science Meeting (Joint with AP-S Symposium), Montreal, QC, Canada.","DOI":"10.23919\/USNC\/URSI49741.2020.9321658"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"4209","DOI":"10.1109\/TWC.2020.2981320","article-title":"Power control based on deep reinforcement learning for spectrum sharing","volume":"19","author":"Zhang","year":"2020","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Yang, P., Li, L., Yin, J., Zhang, H., Liang, W., Chen, W., and Han, Z. (2018, January 16\u201318). Dynamic spectrum access in cognitive radio networks using deep reinforcement learning and evolutionary game. Proceedings of the 2018 IEEE\/CIC International Conference on Communications in China, Beijing, China.","DOI":"10.1109\/ICCChina.2018.8641242"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Xu, Y., Yu, J., Headley, W.C., and Buehrer, R.M. (2018, January 29\u201331). Deep reinforcement learning for dynamic spectrum access in wireless networks. Proceedings of the MILCOM 2018\u20142018 IEEE Military Communications Conference, Los Angeles, CA, USA.","DOI":"10.1109\/MILCOM.2018.8599723"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Yadav, M.A., Li, Y., Fang, G., and Shen, B. (2022, January 6\u20138). Deep Q-network based reinforcement learning for distributed dynamic spectrum access. Proceedings of the 2022 IEEE 2nd International Conference on Computer Communication and Artificial Intelligence (CCAI), Beijing, China.","DOI":"10.1109\/CCAI55564.2022.9807797"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"11208","DOI":"10.1109\/JIOT.2021.3052691","article-title":"A deep reinforcement learning framework for spectrum management in dynamic spectrum access","volume":"8","author":"Song","year":"2021","journal-title":"IEEE Internet Things J."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Xu, Y., Yu, J., and Buehrer, R.M. (2019, January 11\u201314). Cache-enabled dynamic spectrum access via deep recurrent Q-networks with partial observation. Proceedings of the 2019 IEEE International Symposium on Dynamic Spectrum Access Networks, Newark, NJ, USA.","DOI":"10.1109\/DySPAN.2019.8935688"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"4494","DOI":"10.1109\/TWC.2020.2984227","article-title":"The application of deep reinforcement learning to distributed spectrum access in dynamic heterogeneous environments With partial observations","volume":"19","author":"Xu","year":"2020","journal-title":"IEEE Trans. Wirel. Commun."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Xu, Y., Yu, J., and Buehrer, R.M. (2018, January 29\u201331). Dealing with partial observations in dynamic sectrum access: Deep recurrent Q-networks. Proceedings of the MILCOM 2018\u20142018 IEEE Military Communications Conference, Los Angeles, CA, USA.","DOI":"10.1109\/MILCOM.2018.8599697"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Bhandari, S., Ranjan, N., Kim, Y., and Kim, H. (2022, January 6\u20139). Deep reinforcement learning for dynamic spectrum access in the multi-channel wireless local area networks. Proceedings of the 2022 International Conference on Electronics, Information, and Communication(ICEIC), Jeju, Republic of Korea.","DOI":"10.1109\/ICEIC54506.2022.9748733"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Cong, Q., and Lang, W. (2021, January 9\u201311). Deep multi-user reinforcement learning for centralized dynamic multichannel access. Proceedings of the 2021 6th International Conference on Intelligent Computing and Signal Processing (ICSP), Xi\u2019an, China.","DOI":"10.1109\/ICSP51882.2021.9408908"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"15554","DOI":"10.1109\/JSEN.2021.3056463","article-title":"Intelligent dynamic spectrum access using deep reinforcement learning for VANETs","volume":"21","author":"Wang","year":"2021","journal-title":"IEEE Sens. J."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"464","DOI":"10.1109\/TCCN.2020.2982895","article-title":"Deep reinforcement learning for dynamic spectrum sensing and aggregation in multi-channel wireless networks","volume":"6","author":"Li","year":"2020","journal-title":"IEEE Trans. Cogn. Commun. Netw."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Jiang, W., Yu, W., Wang, W., and Huang, T. (2022). Multi-agent reinforcement learning for joint cooperative spectrum sensing and channel access in cognitive UAV networks. Sensors, 2.","DOI":"10.3390\/s22041651"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Maulik, S., Roy, R., De, A., and Bhatttacharya, A. (2012, January 22\u201325). Online dynamic resource allocation in interference temperature constrained cognitive radio network using reinforcement learning. Proceedings of the 2012 International Conference on Signal Processing and Communications, Bangalore, India.","DOI":"10.1109\/SPCOM.2012.6290043"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"4244","DOI":"10.1109\/TII.2021.3113949","article-title":"Reinforcement-learning-based dynamic spectrum access for software-defined cognitive industrial internet of things","volume":"18","author":"Liu","year":"2022","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Janiar, S.B., and Pourahmadi, V. (2021, January 9\u201312). Deep reinforcement learning for fair distributed dynamic spectrum access in wireless networks. Proceedings of the 2021 IEEE 18th Annual Consumer Communications & Networking Conference (CCNC), Las Vegas, NV, USA.","DOI":"10.1109\/CCNC49032.2021.9369536"},{"key":"ref_38","unstructured":"Meinil\u00e4, J., Ky\u00f6sti, P., J\u00e4ms\u00e4, T., and Hentil\u00e4, L. (2009). Radio Technologies and Concepts for IMT-Advanced, Wiley."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Couillard, G., Dahman, G., Poitau, G., and Gagnon, F. (2019, January 5\u20138). Quantifying range extension capability of MIMO: A simulation study based on WINNER II model. Proceedings of the 2019 IEEE Canadian Conference of Electrical and Computer Engineering (CCECE), Edmonton, AB, Canada.","DOI":"10.1109\/CCECE.2019.8861854"},{"key":"ref_40","unstructured":"Sutton, R.S., and Andrew, G.B. (2018). Reinforcement Learning: An Introduction, MIT Press."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_42","unstructured":"Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. (2014). Empirical evaluation of gated recurrent neural networkson sequence modeling. arXiv."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"1550","DOI":"10.1109\/5.58337","article-title":"Backpropagation through time: What it does and how to do it","volume":"78","author":"Werbos","year":"1990","journal-title":"Proc. IEEE"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1109\/LCOMM.2010.01.091494","article-title":"On exploiting idle channels in opportunistic multichannel ALOHA","volume":"1","author":"To","year":"2010","journal-title":"IEEE Commun. Lett."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/5\/2622\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,1,20]],"date-time":"2025-01-20T16:38:56Z","timestamp":1737391136000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/5\/2622"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,27]]},"references-count":45,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2023,3]]}},"alternative-id":["s23052622"],"URL":"https:\/\/doi.org\/10.3390\/s23052622","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2023,2,27]]}}}