Abstract
Integration of large-scale renewable energy sources is the future of smart grids. However, weather conditions significantly affect the generation of renewable energy sources, which causes supply-side uncertainties. Even with the current state-of-the-art technologies, weather conditions, solar radiance, etc., can be predicted nearly accurately, only a few minutes early. Existing intelligent grid optimization methods used to solve Optimal Power Flow (OPF) consider day-ahead planning for congestion management and generator rescheduling. Current OPF-based methods suffer many drawbacks for the large grid, like longer computation time and local convergence issues, and are computationally expensive. This drawback makes existing OPF methods unsuitable for the congestion management problem when considering supply-side uncertainties, mainly due to the large-scale integration of renewable energy sources. Over the years, Reinforcement Learning (RL) methods have gained popularity for solving constrained optimization tasks at scale. However, even if the existing Deep-RL (DRL) based policies perform best for large and complex tasks, the lack of interpretability due to using deep neural networks at their core makes them not a trustworthy solution when safety constraints like congestion-free operation are considered. Therefore, to ensure that DRL policies are used safely in critical infrastructures like smart grids, this paper proposes a centralized barrier-penalty-based DRL method that utilizes the Adaptive Safety Shield framework as an extra layer of safety. The DRL-based policy is trained to find the economically optimal grid state while considering the barrier penalty value to reduce branch loads. Besides, the Adaptive Safety Shield is trained to learn the congestion criteria and guide the RL in avoiding such scenarios during exploration. We have also empirically shown that the proposed method has demonstrated similar or better performance than the current OPF-based methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Abbeel, P., Coates, A., Ng, A.Y.: Autonomous helicopter aerobatics through apprenticeship learning. Int. J. Robot. Res. (IJRR) 29 (2010)
Achiam, J., Held, D., Tamar, A., Abbeel, P.: Constrained policy optimization. In: International Conference on Machine Learning (2017)
Alazemi, T., Darwish, M., Radi, M.: Renewable energy sources integration via machine learning modelling: a systematic literature review. Heliyon 10(4), e26088 (2024). https://doi.org/10.1016/j.heliyon.2024.e26088
Alshiekh, M., Bloem, R., Ehlers, R., Könighofer, B., Niekum, S., Topcu, U.: Safe reinforcement learning via shielding. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., Mané, D.: Concrete problems in AI safety. arXiv preprint arXiv:1606.06565 (2016)
Berkenkamp, F., Moriconi, R., Schoellig, A.P., Krause, A.: Safe learning of regions of attraction for uncertain, nonlinear systems with Gaussian processes. In: 2016 IEEE 55th Conference on Decision and Control. CDC 2016, pp. 4661–4666 (2016). https://doi.org/10.1109/CDC.2016.7798979
Bhattacharya, K., Bollen, M., E. Daalder, J.: Operation of Restructured Power Systems (2001). https://doi.org/10.1007/978-1-4615-1465-7
Center, S.L.D.: Power maps all regions 2013. https://www.coursehero.com/file/65794401/Power-maps-All-Regions-2013pdf/
Chow, Y., Nachum, O., Duenez-Guzman, E., Ghavamzadeh, M.: A Lyapunov-based approach to safe reinforcement learning. arXiv preprint arXiv:1805.07708 (2018)
Christie, R.D., Wollenberg, B.F., Wangensteen, I.: Transmission management in the deregulated environment. Proc. IEEE 88(2), 170–195 (2000)
Dey, S., Dasgupta, P., Gangopadhyay, B.: Safety augmentation in decision trees. In: Proceedings of the Workshop on Artificial Intelligence Safety 2020 co-located with the 29th International Joint Conference on Artificial Intelligence and the 17th Pacific Rim International Conference on Artificial Intelligence (IJCAI-PRICAI 2020), Yokohama, Japan, January 2021. CEUR Workshop Proceedings, vol. 2640. CEUR-WS.org (2020)
Dey, S., Mujumdar, A., Dasgupta, P., Dey, S.: Adaptive safety shields for reinforcement learning-based cell shaping. IEEE Trans. Netw. Serv. Manag. 19(4), 5034–5043 (2022). https://doi.org/10.1109/TNSM.2022.3194566
Di Castro, D., Tamar, A., Mannor, S.: Policy gradients with variance related risk criteria. arXiv preprint arXiv:1206.6404 (2012)
Elsaraiti, M., Merabet, A.: Solar power forecasting using deep learning techniques. IEEE Access 10, 31692–31698 (2022)
Esfahani, M.M., Yousefi, G.R.: Real time congestion management in power systems considering quasi-dynamic thermal rating and congestion clearing time. IEEE Trans. Industr. Inf. 12(2), 745–754 (2016)
Feghhi, S., Aumayr, E., Vannella, F., Hakim, E.A., Iakovidis, G.: Safe reinforcement learning for antenna tilt optimisation using shielding and multiple baselines. arXiv preprint arXiv:2012.01296 (2020)
Fulton, N., Platzer, A.: Safe reinforcement learning via formal methods: toward safe control through proof and learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)
Garcıa, J., Fernández, F.: A comprehensive survey on safe reinforcement learning. J. Mach. Learn. Res. 16(1), 1437–1480 (2015)
Geibel, P., Wysotzki, F.: Risk-sensitive reinforcement learning applied to control under constraints. J. Artif. Intell. Res. 24, 81–108 (2005). https://doi.org/10.1613/jair.1666
Glavic, M., Fonteneau, R., Ernst, D.: Reinforcement learning for electric power system decision and control: past considerations and perspectives. IFAC-PapersOnLine 50(1), 6918–6927 (2017). https://doi.org/10.1016/j.ifacol.2017.08.1217, 20th IFAC World Congress
HUB, E.S.: Photovoltaic geographical information system (pvgis) (2019). https://ec.europa.eu/jrc/en/pvgis
Imthias Ahamed, T., Nagendra Rao, P., Sastry, P.: A reinforcement learning approach to automatic generation control. Electr. Power Syst. Res. 63(1), 9–26 (2002). https://doi.org/10.1016/S0378-7796(02)00088-3
Government of India, M.o.N., Energy, R.: Current status: Ministry of new and renewable energy, Government of India. https://mnre.gov.in/solar/current-status/
Liu, Z., et al.: Artificial intelligence powered large-scale renewable integrations in multi-energy systems for carbon neutrality transition: challenges and future perspectives. Energy AI 10, 100195 (2022). https://doi.org/10.1016/j.egyai.2022.100195
Maiti, S., Balabhaskara, A., Adhikary, S., Koley, I., Dey, S.: Targeted attack synthesis for smart grid vulnerability analysis. In: Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pp. 2576–2590 (2023)
Mnih, V., et al.: Playing Atari with deep reinforcement learning. arXiv abs/1312.5602 (2013)
van Otterlo, M., Wiering, M.A.: Markov Decision Processes: Concepts and Algorithms (2012)
Perkins, T.J., Barto, A.G.: Lyapunov design for safe reinforcement learning. J. Mach. Learn. Res. 3(Dec), 803–832 (2002)
Ray, A., Achiam, J., Amodei, D.: Benchmarking Safe Exploration in Deep Reinforcement Learning (2019)
Stott, B., Alsac, O., Monticelli, A.J.: Security analysis and optimization. Proc. IEEE 75(12), 1623–1644 (1987)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction (2018)
Sutton, R.S., Barto, A.G., et al.: Introduction to Reinforcement Learning, vol. 135. MIT Press, Cambridge (1998)
Talukdar, B., Sinha, A., Mukhopadhyay, S., Bose, A.: A computationally simple method for cost-efficient generation rescheduling and load shedding for congestion management. Int. J. Electric. Power Energy Syst. 27(5–6), 379–388 (2005)
Verma, P., Dasgupta, P., Chakraborty, C.: Ml-assisted real time congestion mitigation under supply-side uncertainties. In: 2021 IEEE PES Innovative Smart Grid Technologies - Asia (ISGT Asia), pp. 1–5 (2021). https://doi.org/10.1109/ISGTAsia49270.2021.9715620
Vlachogiannis, J., Hatziargyriou, N.: Reinforcement learning for reactive power control. IEEE Trans. Power Syst. 19(3), 1317–1325 (2004). https://doi.org/10.1109/TPWRS.2004.831259
Xu, Y., Zhang, W., Liu, W., Ferrese, F.: Multiagent-based reinforcement learning for optimal reactive power dispatch. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 42(6), 1742–1751 (2012). https://doi.org/10.1109/TSMCC.2012.2218596
Yu, T., Zhou, B., Chan, K.W., Yuan, Y., Yang, B., Wu, Q.H.: R(\(\lambda \)) imitation learning for automatic generation control of interconnected power grids. Automatica 48(9), 2130–2136 (2012). https://doi.org/10.1016/j.automatica.2012.05.043
Yu, T., Zhou, B., Chan, K.W., Chen, L., Yang, B.: Stochastic optimal relaxed automatic generation control in non-Markov environment based on multi-step q(\(\lambda \)) learning. IEEE Trans. Power Syst. 26(3), 1272–1282 (2011). https://doi.org/10.1109/TPWRS.2010.2102372
Zarrabian, S., Belkacemi, R., Babalola, A.A.: Reinforcement learning approach for congestion management and cascading failure prevention with experimental application. Electr. Power Syst. Res. 141(C), 179–190 (2016). https://doi.org/10.1016/j.epsr.2016.06.041
Zhao, N., You, F.: Sustainable power systems operations under renewable energy induced disjunctive uncertainties via machine learning-based robust optimization. Renew. Sustain. Energy Rev. 161, 112428 (2022). https://doi.org/10.1016/j.rser.2022.112428
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Dey, S., Verma, P., Dasgupta, P., Dey, S. (2024). An Adaptive Interpretable Safe-RL Approach for Addressing Smart Grid Supply-Side Uncertainties. In: Calvaresi, D., et al. Explainable and Transparent AI and Multi-Agent Systems. EXTRAAMAS 2024. Lecture Notes in Computer Science(), vol 14847. Springer, Cham. https://doi.org/10.1007/978-3-031-70074-3_7
Download citation
DOI: https://doi.org/10.1007/978-3-031-70074-3_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-70073-6
Online ISBN: 978-3-031-70074-3
eBook Packages: Computer ScienceComputer Science (R0)