{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,8,25]],"date-time":"2024-08-25T11:10:10Z","timestamp":1724584210445},"reference-count":52,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2023,1,18]],"date-time":"2023-01-18T00:00:00Z","timestamp":1674000000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"the Nigerian Petroleum Technology Trust Fund (PTDF) Overseas Scholarship Scheme","award":["18GFC\/PHD\/020"]},{"name":"Paul Sabatier University"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computers"],"abstract":"Q-learning has been primarily used as one of the reinforcement learning (RL) techniques to find the optimal routing path in wireless sensor networks (WSNs). However, for the centralized RL-based routing protocols with a large state space and action space, the baseline Q-learning used to implement these protocols suffers from degradation in the convergence speed, network lifetime, and network energy consumption due to the large number of learning episodes required to learn the optimal routing path. To overcome these limitations, an efficient model-free RL-based technique called Least-Square Policy Iteration (LSPI) is proposed to optimize the network lifetime and energy consumption in WSNs. The resulting designed protocol is a Centralized Routing Protocol for Lifetime and Energy Optimization with a Genetic Algorithm (GA) and LSPI (CRPLEOGALSPI). Simulation results show that the CRPLEOGALSPI has improved performance in network lifetime and energy consumption compared to an existing Centralized Routing Protocol for Lifetime Optimization with GA and Q-learning (CRPLOGARL). This is because the CRPLEOGALSPI chooses a routing path in a given state considering all the possible routing paths, and it is not sensitive to the learning rate. Moreover, while the CRPLOGARL evaluates the optimal policy from the Q-values, the CRPLEOGALSPI updates the Q-values based on the most updated information regarding the network dynamics using weighted functions.<\/jats:p>","DOI":"10.3390\/computers12020022","type":"journal-article","created":{"date-parts":[[2023,1,19]],"date-time":"2023-01-19T07:51:37Z","timestamp":1674114697000},"page":"22","source":"Crossref","is-referenced-by-count":1,"title":["A Centralized Routing for Lifetime and Energy Optimization in WSNs Using Genetic Algorithm and Least-Square Policy Iteration"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"http:\/\/orcid.org\/0000-0003-4618-5023","authenticated-orcid":false,"given":"Elvis","family":"Obi","sequence":"first","affiliation":[{"name":"Computer Science Research Institute, Paul Sabatier University, 31062 Toulouse, France"}]},{"ORCID":"http:\/\/orcid.org\/0000-0003-4164-9597","authenticated-orcid":false,"given":"Zoubir","family":"Mammeri","sequence":"additional","affiliation":[{"name":"Computer Science Research Institute, Paul Sabatier University, 31062 Toulouse, France"}]},{"given":"Okechukwu E.","family":"Ochia","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, University of Calgary, Calgary, AB T2N 1N4, Canada"}]}],"member":"1968","published-online":{"date-parts":[[2023,1,18]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"7333","DOI":"10.1007\/s11227-020-03166-5","article-title":"Deployment techniques in wireless sensor networks: A survey, classification, challenges, and future research issues","volume":"76","author":"Priyadarshi","year":"2020","journal-title":"J. Supercomput."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s11227-013-1021-9","article-title":"Wireless sensor networks: A survey on recent developments and potential synergies","volume":"68","author":"Rawat","year":"2014","journal-title":"J. Supercomput."},{"key":"ref_3","first-page":"1","article-title":"Overview of wireless sensor network","volume":"1","author":"Matin","year":"2012","journal-title":"Wirel. Sens. Netw.-Technol. Protoc."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"8824","DOI":"10.3390\/s91108824","article-title":"Wireless sensor technologies and applications","volume":"9","author":"Xia","year":"2009","journal-title":"Sensors"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Engmann, F., Katsriku, F.A., Abdulai, J.D., Adu-Manu, K.S., and Banaseka, F.K. (2018). Prolonging the lifetime of wireless sensor networks: A review of current techniques. Wirel. Commun. Mob. Comput., 1\u201323.","DOI":"10.1155\/2018\/8035065"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.measurement.2021.108974","article-title":"Routing in wireless sensor networks using machine learning techniques: Challenges and opportunities","volume":"178","author":"Nayak","year":"2021","journal-title":"Measurement"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"2407","DOI":"10.1007\/s11277-019-06993-9","article-title":"Routing in wireless sensor networks using optimization techniques: A survey","volume":"111","author":"Khedr","year":"2020","journal-title":"Wirel. Pers. Commun."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1016\/j.jnca.2018.06.016","article-title":"Software-defined wireless sensor networks: A survey","volume":"119","author":"Mostafaei","year":"2018","journal-title":"J. Netw. Comput. Appl."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Obi, E., Mammeri, Z., and Ochia, O.E. (2021, January 11\u201313). A Lifetime-Aware Centralized Routing Protocol for Wireless Sensor Networks using Reinforcement Learning. Proceedings of the 17th International Conference on Wireless and Mobile Computing, Networking and Communications, Bologna, Italy.","DOI":"10.1109\/WiMob52687.2021.9606390"},{"key":"ref_10","unstructured":"Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT press. [2nd ed.]."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"3175","DOI":"10.1080\/00207160903329699","article-title":"Listing all the minimum spanning trees in an undirected graph","volume":"87","author":"Yamada","year":"2010","journal-title":"Int. J. Comput. Math."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1007\/BF00175354","article-title":"A genetic algorithm tutorial","volume":"4","author":"Whitley","year":"1994","journal-title":"Stat. Comput."},{"key":"ref_13","unstructured":"Obi, E., Mammeri, Z., and Ochia, O.E. (2022, January 16\u201320). Centralized Routing for Lifetime Optimization Using Genetic Algorithm and Reinforcement Learning for WSNs. Proceedings of the 16th International Conference on Sensor Technologies and Applications, Lisbon, Portugal."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1007\/BF00992698","article-title":"Q-learning","volume":"8","author":"Watkins","year":"1992","journal-title":"Mach. Learn."},{"key":"ref_15","first-page":"1107","article-title":"Least-squares policy iteration","volume":"4","author":"Lagoudakis","year":"2003","journal-title":"J. Mach. Learn. Res."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1613\/jair.301","article-title":"Reinforcement learning: A survey","volume":"4","author":"Kaelbling","year":"1996","journal-title":"J. Artif. Intell. Res."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"55916","DOI":"10.1109\/ACCESS.2019.2913776","article-title":"Reinforcement learning based routing in networks: Review and classification of approaches","volume":"7","author":"Mammeri","year":"2019","journal-title":"IEEE Access"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1007\/BF00114723","article-title":"Linear least-squares algorithms for temporal difference learning","volume":"22","author":"Bradtke","year":"1996","journal-title":"Mach. Learn."},{"key":"ref_19","first-page":"671","article-title":"Packet routing in dynamically changing networks: A reinforcement learning approach","volume":"6","author":"Boyan","year":"1993","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Zhang, Y., and Fromherz, M. (2006, January 18\u201320). Constrained flooding: A robust and efficient routing framework for wireless sensor networks. Proceedings of the 20th International Conference on Advanced Information Networking and Applications-Volume 1, Vienna, Austria.","DOI":"10.1109\/AINA.2006.132"},{"key":"ref_21","unstructured":"Maroti, M. (2004, January 18\u201320). Directed flood-routing framework for wireless sensor networks. Proceedings of the ACM\/IFIP\/USENIX International Conference on Distributed Systems Platforms and Open Distributed Processing, Berlin, Germany."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"He, T., Krishnamurthy, S., Stankovic, J.A., Abdelzaher, T., Luo, L., Stoleru, R., Yan, T., Gu, L., Hui, J., and Krogh, B. (2004, January 6\u20139). Energy-efficient surveillance system using wireless sensor networks. Proceedings of the 2nd International Conference on Mobile Systems, Applications, and Services, Boston, MA, USA.","DOI":"10.1145\/990064.990096"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Intanagonwiwat, C., Govindan, R., and Estrin, D. (2000, January 6\u201311). Directed diffusion: A scalable and robust communication paradigm for sensor networks. Proceedings of the 6th Annual International Conference on Mobile Computing and Networking, Boston, MA, USA.","DOI":"10.1145\/345910.345920"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Wang, P., and Wang, T. (2006, January 20\u201322). Adaptive routing for sensor networks using reinforcement learning. Proceedings of the 6th IEEE International Conference on Computer and Information Technology, Seoul, Republic of Korea.","DOI":"10.1109\/CIT.2006.34"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Nurmi, P. (2007, January 16\u201320). Reinforcement learning for routing in ad hoc networks. Proceedings of the 5th IEEE International Symposium on Modeling and Optimization in Mobile, Ad Hoc and Wireless Networks and Workshops, Limassol, Cyprus.","DOI":"10.1109\/WIOPT.2007.4480049"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Dong, S., Agrawal, P., and Sivalingam, K. (2007, January 26\u201330). Reinforcement learning based geographic routing protocol for UWB wireless sensor network. Proceedings of the IEEE Global Telecommunications Conference, Washington, DC, USA.","DOI":"10.1109\/GLOCOM.2007.127"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Karp, B., and Kung, H.T. (2000, January 6\u201311). GPSR: Greedy perimeter stateless routing for wireless networks. Proceedings of the 6th Annual International Conference on Mobile Computing and Networking, Boston MA, USA.","DOI":"10.1145\/345910.345953"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Arroyo-Valles, R., Alaiz-Rodriguez, R., Guerrero-Curieses, A., and Cid-Sueiro, J. (2007, January 3\u20136). Q-probabilistic routing in wireless sensor networks. Proceedings of the IEEE 3rd International Conference on Intelligent Sensors, Sensor Networks and Information, Melbourne, VIC, Australia.","DOI":"10.1109\/ISSNIP.2007.4496810"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Naruephiphat, W., and Usaha, W. (2008, January 11\u201314). Balancing tradeoffs for energy-efficient routing in MANETs based on reinforcement learning. Proceedings of the VTC Spring IEEE Vehicular Technology Conference, Marina Bay, Singapore.","DOI":"10.1109\/VETECS.2008.523"},{"key":"ref_30","unstructured":"F\u00f6rster, A., and Murphy, A.L. (2008, January 11\u201314). Balancing energy expenditure in WSNs through reinforcement learning: A study. Proceedings of the 1st International Workshop on Energy in Wireless Sensor Networks, Santorini Island, Greece."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Hu, T., and Fei, Y. (2008, January 7\u20139). QELAR: A q-learning-based energy-efficient and lifetime-aware routing protocol for underwater sensor networks. Proceedings of the IEEE International Performance, Computing and Communications Conference, Austin, TX, USA.","DOI":"10.1109\/PCCC.2008.4745119"},{"key":"ref_32","unstructured":"Yang, J., Zhang, H., Pan, C., and Sun, W. (2013, January 6\u20139). Learning-based routing approach for direct interactions between wireless sensor network and moving vehicles. Proceedings of the 16th International IEEE Conference on Intelligent Transportation Systems, The Hague, The Netherlands."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Oddi, G., Pietrabissa, A., and Liberati, F. (2014, January 14\u201317). Energy balancing in multi-hop Wireless Sensor Networks: An approach based on reinforcement learning. Proceedings of the 2014 NASA\/ESA IEEE Conference on Adaptive Hardware and Systems, Leicester, UK.","DOI":"10.1109\/AHS.2014.6880186"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Jafarzadeh, S.Z., and Moghaddam, M.H.Y. (2014, January 4\u20137). Design of energy-aware QoS routing protocol in wireless sensor networks using reinforcement learning. Proceedings of the 2014 IEEE 27th Canadian Conference on Electrical and Computer Engineering, Toronto, ON, Canada.","DOI":"10.1109\/CCECE.2014.6900988"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"487","DOI":"10.4028\/www.scientific.net\/AMM.678.487","article-title":"An intelligent routing algorithm in wireless sensor networks based on reinforcement learning","volume":"678","author":"Guo","year":"2014","journal-title":"Appl. Mech. Mater."},{"key":"ref_36","unstructured":"Shah, R.C., and Rabaey, J.M. (2002, January 17\u201321). Energy aware routing for low energy ad hoc sensor networks. Proceedings of the IEEE Wireless Communications and Networking Conference Record, Orlando, FL, USA."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Yessad, S., Tazarart, N., Bakli, L., Medjkoune-Bouallouche, L., and Aissani, D. (2012, January 26\u201328). Balanced energy-efficient routing protocol for WSN. Proceedings of the IEEE International Conference on Communications and Information Technology, Hammamet, Tunisia.","DOI":"10.1109\/ICCITechnol.2012.6285817"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Debowski, B., Spachos, P., and Areibi, S. (2016, January 23\u201325). Q-learning enhanced gradient-based routing for balancing energy consumption in WSNs. Proceedings of the IEEE 21st International Workshop on Computer Aided Modelling and Design of Communication Links and Networks, Toronto, ON, Canada.","DOI":"10.1109\/CAMAD.2016.7790324"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"5061","DOI":"10.1007\/s11277-016-3729-3","article-title":"MRL-SCSO: Multi-agent reinforcement learning-based self-configuration and self-optimization protocol for unattended wireless sensor networks","volume":"96","author":"Renold","year":"2017","journal-title":"Wirel. Pers. Commun."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Gnawali, O., Fonseca, R., Jamieson, K., Moss, D., and Levis, P. (2009, January 4\u20136). Collection tree protocol. Proceedings of the 7th ACM Conference on Embedded Networked Sensor Systems, Berkeley, CA, USA.","DOI":"10.1145\/1644038.1644040"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1177\/1550147719833541","article-title":"Optimizing the lifetime of wireless sensor networks via reinforcement-learning-based routing","volume":"15","author":"Guo","year":"2019","journal-title":"Int. J. Distrib. Sens. Netw."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Bouzid, S.E., Serrestou, Y., Raoof, K., and Omri, M.N. (2020, January 2\u20135). Efficient routing protocol for wireless sensor network based on reinforcement learning. Proceedings of the 5th IEEE International Conference on Advanced Technologies for Signal and Image Processing, Sousse, Tunisia.","DOI":"10.1109\/ATSIP49331.2020.9231883"},{"key":"ref_43","first-page":"1","article-title":"Analyzing the energy efficient path in Wireless Sensor Network using Machine Learning","volume":"10","author":"Sapkota","year":"2021","journal-title":"ADBU J. Eng. Technol."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"2","DOI":"10.1109\/TNET.2002.808417","article-title":"Directed diffusion for wireless sensor networking","volume":"11","author":"Intanagonwiwat","year":"2003","journal-title":"IEEE\/ACM Trans. Netw."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Mutombo, V.K., Shin, S.Y., and Hong, J. (2021, January 22\u201326). EBR-RL: Energy balancing routing protocol based on reinforcement learning for WSN. Proceedings of the 36th Annual ACM Symposium on Applied Computing, Virtual Event.","DOI":"10.1145\/3412841.3442063"},{"key":"ref_46","unstructured":"Gibbons, A. (1985). Algorithmic Graph Theory, Cambridge University Press."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"1389","DOI":"10.1002\/j.1538-7305.1957.tb01515.x","article-title":"Shortest connection networks and some generalizations","volume":"36","author":"Prim","year":"1957","journal-title":"Bell Syst. Tech. J."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1090\/S0002-9939-1956-0078686-7","article-title":"On the shortest spanning subtree of a graph and the traveling salesman problem","volume":"7","author":"Kruskal","year":"1956","journal-title":"Proc. Am. Math. Soc."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"377","DOI":"10.1007\/s10586-017-0868-6","article-title":"Optimizing the minimum spanning tree-based extracted clusters using evolution strategy","volume":"21","author":"Halim","year":"2018","journal-title":"Clust. Comput."},{"key":"ref_50","unstructured":"de Almeida, T.A., Yamakami, A., and Takahashi, M.T. (2005, January 28\u201330). An evolutionary approach to solve minimum spanning tree problem with fuzzy parameters. Proceedings of the IEEE International Conference on Computational Intelligence for Modelling, Control and Automation and International Conference on Intelligent Agents, Web Technologies and Internet Commerce, Washington, DC, USA."},{"key":"ref_51","unstructured":"Almeida, T.A., Souza, V.N., Prado, F.M.S., Yamakami, A., and Takahashi, M.T. (2005, January 26\u201328). A genetic algorithm to solve minimum spanning tree problem with fuzzy parameters using possibility measure. Proceedings of the IEEE NAFIPS Annual Meeting of the North American Fuzzy Information Processing Society, Detroit, MI, USA."},{"key":"ref_52","unstructured":"Hagberg, A., Swart, P., and Daniel, S.C. (2008, January 19\u201324). Exploring network structure, dynamics, and function using NetworkX. Proceedings of the 8th SCIPY Conference, Pasadena, CA, USA."}],"container-title":["Computers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-431X\/12\/2\/22\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,25]],"date-time":"2024-08-25T10:20:28Z","timestamp":1724581228000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-431X\/12\/2\/22"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,18]]},"references-count":52,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2023,2]]}},"alternative-id":["computers12020022"],"URL":"https:\/\/doi.org\/10.3390\/computers12020022","relation":{},"ISSN":["2073-431X"],"issn-type":[{"type":"electronic","value":"2073-431X"}],"subject":[],"published":{"date-parts":[[2023,1,18]]}}}