{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,4,4]],"date-time":"2025-04-04T10:06:19Z","timestamp":1743761179257,"version":"3.37.3"},"reference-count":23,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,12,1]],"date-time":"2020-12-01T00:00:00Z","timestamp":1606780800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,12,10]],"date-time":"2020-12-10T00:00:00Z","timestamp":1607558400000},"content-version":"vor","delay-in-days":9,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001804","name":"Canada Research Chairs","doi-asserted-by":"publisher","award":["Tier 1"],"id":[{"id":"10.13039\/501100001804","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61502067"],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100007957","name":"Chongqing Municipal Education Commission","doi-asserted-by":"publisher","award":["KJZD-K201800603"],"id":[{"id":"10.13039\/501100007957","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Doctoral Candidate Innovative Talent Project of CQUPT","award":["BYJS2017003"]},{"DOI":"10.13039\/501100004543","name":"China Scholarship Council","doi-asserted-by":"publisher","award":["201908500144"],"id":[{"id":"10.13039\/501100004543","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Wireless Com Network"],"published-print":{"date-parts":[[2020,12]]},"abstract":"Abstract<\/jats:title>Ultra-reliable and low-latency communication (URLLC) in mobile networks is still one of the core solutions that require thorough research in 5G and beyond. With the vigorous development of various emerging URLLC technologies, resource shortages will soon occur even in mmWave cells with rich spectrum resources. As a result of the large radio resource space of mmWave, traditional real-time resource scheduling decisions can cause serious delays. Consequently, we investigate a delay minimization problem with the spectrum and power constraints in the mmWave hybrid access network. To reduce the delay caused by high load and radio resource shortage, a hybrid spectrum and power resource allocation scheme based on reinforcement learning (RL) is proposed. We compress the state space and the action space by temporarily dumping and decomposing the action. The multipath deep neural network and policy gradient method are used, respectively, as the approximater and update method of the parameterized policy. The experimental results reveal that the RL-based hybrid spectrum and the power resource allocation scheme eventually converged after a limited number of iterative learnings. Compared with other schemes, the RL-based scheme can effectively guarantee the URLLC delay constraint when the load does not exceed 130%.<\/jats:p>","DOI":"10.1186\/s13638-020-01872-5","type":"journal-article","created":{"date-parts":[[2020,12,10]],"date-time":"2020-12-10T19:03:19Z","timestamp":1607626999000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Reinforcement learning-based hybrid spectrum resource allocation scheme for the high load of URLLC services"],"prefix":"10.1186","volume":"2020","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4635-2952","authenticated-orcid":false,"given":"Qian","family":"Huang","sequence":"first","affiliation":[]},{"given":"Xianzhong","family":"Xie","sequence":"additional","affiliation":[]},{"given":"Mohamed","family":"Cheriet","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2020,12,10]]},"reference":[{"issue":"5","key":"1872_CR1","doi-asserted-by":"publisher","first-page":"3187","DOI":"10.1109\/TCOMM.2020.2971486","volume":"68","author":"E Basar","year":"2020","unstructured":"E. Basar, Reconfigurable intelligent surface-based index modulation: a new beyond MIMO paradigm for 6G. IEEE Trans. Commun. 68(5), 3187\u20133196 (2020)","journal-title":"IEEE Trans. Commun."},{"issue":"3","key":"1872_CR2","doi-asserted-by":"publisher","first-page":"124","DOI":"10.1109\/MWC.2018.1700294","volume":"25","author":"H Ji","year":"2018","unstructured":"H. Ji, S. Park, J. Yeo et al., Ultra-reliable and low-latency communications in 5G downlink: physical layer aspects. IEEE Wirel. Commun. 25(3), 124\u2013130 (2018)","journal-title":"IEEE Wirel. Commun."},{"issue":"2","key":"1872_CR3","doi-asserted-by":"publisher","first-page":"94","DOI":"10.1109\/MVT.2019.2903657","volume":"14","author":"D Feng","year":"2019","unstructured":"D. Feng, C. She, K. Ying et al., Toward ultrareliable low-latency communications: typical scenarios, possible solutions, and open issues. IEEE Veh. Technol. Mag. 14(2), 94\u2013102 (2019)","journal-title":"IEEE Veh. Technol. Mag."},{"issue":"4","key":"1872_CR4","doi-asserted-by":"publisher","first-page":"30","DOI":"10.1109\/MNET.2019.1800424","volume":"33","author":"Q Huang","year":"2019","unstructured":"Q. Huang, X. Xie, H. Tang et al., Machine-learning-based cognitive spectrum assignment for 5G URLLC applications. IEEE Netw. 33(4), 30\u201335 (2019)","journal-title":"IEEE Netw."},{"key":"1872_CR5","doi-asserted-by":"publisher","first-page":"1023","DOI":"10.1109\/TVT.2019.2954462","volume":"69","author":"H Yang","year":"2019","unstructured":"H. Yang, K. Zheng, L. Zhao et al., Twin-timescale radio resource management for ultra-reliable and low-latency vehicular networks. IEEE Trans. Veh. Technol. 69, 1023\u20131036 (2019)","journal-title":"IEEE Trans. Veh. Technol."},{"key":"1872_CR6","doi-asserted-by":"publisher","DOI":"10.1109\/lwc.2019.2956536","author":"Z Wang","year":"2019","unstructured":"Z. Wang, T. Lv, Z. Lin et al., Outage performance of URLLC NOMA systems with wireless power transfer. IEEE Wirel. Commun. Lett. (2019). https:\/\/doi.org\/10.1109\/lwc.2019.2956536","journal-title":"IEEE Wirel. Commun. Lett."},{"unstructured":"3GPP, Study on physical layer enhancements for NR ultra-reliable and low latency case (URLLC), document TR38.824 V16.0.0 (2019)","key":"1872_CR7"},{"key":"1872_CR8","doi-asserted-by":"publisher","first-page":"2727","DOI":"10.1109\/JSAC.2019.2947941","volume":"37","author":"X Zhang","year":"2019","unstructured":"X. Zhang, J. Wang, H.V. Poor, Heterogeneous statistical-QoS driven resource allocation over mmWave massive-MIMO based 5G mobile wireless networks in the non-asymptotic regime. IEEE J. Sel. Areas Commun. 37, 2727\u20132743 (2019)","journal-title":"IEEE J. Sel. Areas Commun."},{"key":"1872_CR9","doi-asserted-by":"publisher","first-page":"2382","DOI":"10.1109\/TWC.2020.2964543","volume":"19","author":"C Zhao","year":"2020","unstructured":"C. Zhao, Y. Cai, A. Liu et al., Mobile edge computing meets mmWave communications: joint beamforming and resource allocation for system delay minimization. IEEE Trans. Wirel. Commun. 19, 2382\u20132396 (2020)","journal-title":"IEEE Trans. Wirel. Commun."},{"issue":"7","key":"1872_CR10","doi-asserted-by":"publisher","first-page":"92","DOI":"10.1109\/MCOM.2019.1800663","volume":"57","author":"X Lu","year":"2019","unstructured":"X. Lu, V. Petrov, D. Moltchanov et al., 5G-U: Conceptualizing integrated utilization of licensed and unlicensed spectrum for future IoT. IEEE Commun. Mag. 57(7), 92\u201398 (2019)","journal-title":"IEEE Commun. Mag."},{"issue":"7","key":"1872_CR11","doi-asserted-by":"publisher","first-page":"32","DOI":"10.23919\/JCC.2019.07.003","volume":"16","author":"R Xie","year":"2019","unstructured":"R. Xie, J. Wu, R. Wang et al., A game theoretic approach for hierarchical caching resource sharing in 5G networks with virtualization. China Commun. 16(7), 32\u201348 (2019)","journal-title":"China Commun."},{"issue":"3","key":"1872_CR12","doi-asserted-by":"publisher","first-page":"677","DOI":"10.1109\/LWC.2018.2882445","volume":"8","author":"F Zhou","year":"2019","unstructured":"F. Zhou, W. Li, L. Meng et al., Capacity enhancement for hotspot area in 5G cellular networks using mmWave aerial base station. IEEE Wirel. Commun. Lett. 8(3), 677\u2013680 (2019)","journal-title":"IEEE Wirel. Commun. Lett."},{"issue":"3","key":"1872_CR13","doi-asserted-by":"publisher","first-page":"3027","DOI":"10.1109\/TVT.2019.2893928","volume":"68","author":"H Huang","year":"2019","unstructured":"H. Huang, Y. Song, J. Yang et al., Deep-learning-based millimeter-wave massive MIMO for hybrid precoding. IEEE Trans. Veh. Technol. 68(3), 3027\u20133032 (2019)","journal-title":"IEEE Trans. Veh. Technol."},{"issue":"6","key":"1872_CR14","doi-asserted-by":"publisher","first-page":"5871","DOI":"10.1109\/TVT.2019.2907682","volume":"68","author":"C Qiu","year":"2019","unstructured":"C. Qiu, H. Yao, F.R. Yu et al., Deep q-learning aided networking, caching, and computing resources allocation in software-defined satellite-terrestrial networks. IEEE Trans. Veh. Technol. 68(6), 5871\u20135883 (2019)","journal-title":"IEEE Trans. Veh. Technol."},{"key":"1872_CR15","doi-asserted-by":"publisher","first-page":"74412","DOI":"10.1109\/ACCESS.2019.2920662","volume":"7","author":"K Shimotakahara","year":"2019","unstructured":"K. Shimotakahara, M. Elsayed, K. Hinzer et al., High-reliability multi-agent Q-learning-based scheduling for D2D microgrid communications. IEEE Access 7, 74412\u201374421 (2019)","journal-title":"IEEE Access"},{"doi-asserted-by":"crossref","unstructured":"W. AlSobhi, A.H. Aghvami, QoS-Aware resource allocation of two-tier HetNet: a Q-learning approach, in 26th International Conference on Telecommunications (ICT) (IEEE, 2019), pp. 330\u2013334.","key":"1872_CR16","DOI":"10.1109\/ICT.2019.8798829"},{"doi-asserted-by":"crossref","unstructured":"A.T.Z. Kasgari, W. Saad, Model-free ultra reliable low latency communication (URLLC): a deep reinforcement learning framework, in ICC 2019\u20132019 IEEE International Conference on Communications (ICC) (IEEE, 2019), pp. 1\u20136.","key":"1872_CR17","DOI":"10.1109\/ICC.2019.8761721"},{"doi-asserted-by":"crossref","unstructured":"N.B. Khalifa, M. Assaad, M. Debbah, Risk-sensitive reinforcement learning for urllc traffic in wireless networks, in 2019 IEEE Wireless Communications and Networking Conference (WCNC) (IEEE, 2019), p. 1\u20137.","key":"1872_CR18","DOI":"10.1109\/WCNC.2019.8885631"},{"issue":"5","key":"1872_CR19","doi-asserted-by":"publisher","first-page":"4157","DOI":"10.1109\/TVT.2018.2890686","volume":"68","author":"H Yang","year":"2019","unstructured":"H. Yang, X. Xie, M. Kadoch, Intelligent resource management based on reinforcement learning for ultra-reliable and low-latency IoV communication networks. IEEE Trans. Veh. Technol. 68(5), 4157\u20134169 (2019)","journal-title":"IEEE Trans. Veh. Technol."},{"issue":"4","key":"1872_CR20","doi-asserted-by":"publisher","first-page":"2268","DOI":"10.1109\/TWC.2019.2963667","volume":"19","author":"X Chen","year":"2020","unstructured":"X. Chen, C. Wu, T. Chen et al., Age of information aware radio resource management in vehicular networks: a proactive deep reinforcement learning perspective. IEEE Trans. Wirel. Commun. 19(4), 2268\u20132281 (2020)","journal-title":"IEEE Trans. Wirel. Commun."},{"unstructured":"R.S. Sutton, D.A. McAllester, S.P. Singh, et al. Policy gradient methods for reinforcement learning with function approximation, in Advances in Neural Information Processing Systems (2000), p. 1057\u20131063","key":"1872_CR21"},{"doi-asserted-by":"crossref","unstructured":"Ciosek K, Whiteson S. Expected policy gradients, in Thirty-Second AAAI Conference on Artificial Intelligence (2018)","key":"1872_CR22","DOI":"10.1609\/aaai.v32i1.11607"},{"issue":"4","key":"1872_CR23","doi-asserted-by":"publisher","first-page":"455","DOI":"10.1145\/2740070.2626334","volume":"44","author":"R Grandl","year":"2014","unstructured":"R. Grandl, G. Ananthanarayanan, S. Kandula et al., Multi-resource packing for cluster schedulers. ACM SIGCOMM Comput. Commun. Rev. 44(4), 455\u2013466 (2014)","journal-title":"ACM SIGCOMM Comput. Commun. Rev."}],"container-title":["EURASIP Journal on Wireless Communications and Networking"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s13638-020-01872-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1186\/s13638-020-01872-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s13638-020-01872-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,4]],"date-time":"2022-12-04T01:33:04Z","timestamp":1670117584000},"score":1,"resource":{"primary":{"URL":"https:\/\/jwcn-eurasipjournals.springeropen.com\/articles\/10.1186\/s13638-020-01872-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12]]},"references-count":23,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,12]]}},"alternative-id":["1872"],"URL":"https:\/\/doi.org\/10.1186\/s13638-020-01872-5","relation":{},"ISSN":["1687-1499"],"issn-type":[{"type":"electronic","value":"1687-1499"}],"subject":[],"published":{"date-parts":[[2020,12]]},"assertion":[{"value":"13 October 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 December 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 December 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare that they have no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"250"}}