Safe optimal robust control of nonlinear systems with asymmetric input constraints using reinforcement learning | Applied Intelligence Skip to main content
Springer Nature Link
Log in

Safe optimal robust control of nonlinear systems with asymmetric input constraints using reinforcement learning

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

External disturbances and asymmetric input constraints may cause a major problem to the optimal control of the system. Aiming at such problem, this article presents a safe and optimal robust control method based on adaptive dynamic programming (ADP) to ensure the system operated in a safe region and with the optimal performance. Initially, a novel nonquadratic form cost function is imported for the system to address the asymmetric input constraints. Then, to ensure the safety of the system, a control barrier function (CBF) is appended to the cost function to penalize the unsafe behavior. And a damping factor is also introduced to the CBF to balance safety and optimality. Finally, one single critic network is utilized to simplify the complex computational steps, which is different from the traditional actor-critic networks to address the Hamilton-Jacobi-Bellman Equation (HJBE) for obtaining the optimal neural controller. Additionally, based on Lyapunov method, all signals in the closed-loop system are proven to be uniformly ultimately bounded (UUB). At last, the experimental results confirm the effectiveness of the designed approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data Availability Statement

The authors can confirm that all relevant data are included in the article.

References

  1. Liu D, Xue S, Zhao B, Luo B, Wei Q (2020) Adaptive dynamic programming for control: a survey and recent advances. IEEE Transactions on Systems, Man, and Cybernetics: Systems 51(1):142–160

    Article  Google Scholar 

  2. Wang D, Ha M, Qiao J (2020) Data-driven iterative adaptive critic control toward an urban wastewater treatment plant. IEEE Trans Industr Electron 68(8):7362–7369

    Article  Google Scholar 

  3. Wang D, Qiao J, Cheng L (2020) An approximate neuro-optimal solution of discounted guaranteed cost control design. IEEE Transactions on Cybernetics 52(1):77–86

    Article  Google Scholar 

  4. Wang D, Li X, Zhao M, Qiao J (2023) Adaptive critic control design with knowledge transfer for wastewater treatment applications. IEEE Transactions on industrial informatics

  5. Wei Q, Zhou T, Lu J, Liu Y, Su S, Xiao J (2023) Continuous-time stochastic policy iteration of adaptive dynamic programming. IEEE Transactions on Systems, Man, and Cybernetics: Systems

  6. Qin C, Wang J, Zhu H, Zhang J, Hu S, Zhang D (2022) Neural network-based safe optimal robust control for affine nonlinear systems with unmatched disturbances. Neurocomputing 506:228–239

    Article  Google Scholar 

  7. Song R, Liu L, Xia L, Lewis FL (2022) Online optimal event-triggered h\(\infty \) control for nonlinear systems with constrained state and input. IEEE Transactions on Systems, Man, and Cybernetics: Systems 53(1):131–141

    Article  Google Scholar 

  8. Bellman R (1966) Dynamic programming. Science 153(3731):34–37

    Article  Google Scholar 

  9. Werbos P (1977) Advanced forecasting methods for global crisis warning and models of intelligence. Gen Syst Yearb 25–38

  10. Chauhan S, Singh M, Aggarwal AK (2023) Designing of optimal digital IIR filter in the multi-objective framework using an evolutionary algorithm. Eng Appl Artif Intell 119:105803

  11. Chauhan S, Singh M, Aggarwal AK (2021) Experimental analysis of effect of tuning parameters on the performance of diversity-driven multi-parent evolutionary algorithm. In:2021 IEEE 2Nd International conference on electrical power and energy systems (ICEPES). IEEE, pp 1–6

  12. Chauhan S, Singh M, Aggarwal AK (2023) Investigative analysis of different mutation on diversity-driven multi-parent evolutionary algorithm and its application in area coverage optimization of WSN. Soft Comput 1–27

  13. Wang N, Gao Y, Yang C, Zhang X (2022) Reinforcement learning-based finite-time tracking control of an unknown unmanned surface vehicle with input constraints. Neurocomputing 484:26–37

    Article  Google Scholar 

  14. Sun J, Zhang H, Yan Y, Xu S, Fan X (2021) Optimal regulation strategy for nonzero-sum games of the immune system using adaptive dynamic programming. IEEE Transactions on cybernetics

  15. Zhang H, Wang H, Niu B, Zhang L, Ahmad AM (2021) Sliding-mode surface-based adaptive actor-critic optimal control for switched nonlinear systems with average dwell time. Inf Sci 580:756–774

    Article  MathSciNet  Google Scholar 

  16. Vamvoudakis KG, Lewis FL (2010) Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem. Automatica 46(5):878–888

    Article  MathSciNet  Google Scholar 

  17. Li D, Dong J (2022) Fuzzy control based on reinforcement learning and subsystem error derivatives for strict-feedback systems with an observer. IEEE Transactions on Fuzzy Systems

  18. Li D, Dong J (2023) Fuzzy weight-based reinforcement learning for event-triggered optimal backstepping control of fractional-order nonlinear systems. IEEE Transactions on Fuzzy Systems

  19. Huang X, Dong J (2020) ADP-based robust resilient control of partially unknown nonlinear systems via cooperative interaction design. IEEE Transactions on Systems, Man, and Cybernetics: Systems 51(12):7466–7474

    Article  Google Scholar 

  20. Wang K, Mu C, Ni Z, Liu D (2023) Safe reinforcement learning and adaptive optimal control with applications to obstacle avoidance problem. IEEE Transactions on Automation Science and Engineering

  21. Farzanegan B, Jagannathan S (2023) Continual reinforcement learning formulation for zero-sum game-based constrained optimal tracking. IEEE Transactions on Systems, Man, and Cybernetics: Systems

  22. Marvi Z, Kiumarsi B (2021) Safe reinforcement learning: a control barrier function optimization approach. Int J Robust Nonlinear Control 31(6):1923–1940

    Article  MathSciNet  Google Scholar 

  23. Qin C, Zhang Z, Shang Z, Zhang J, Zhang D (2023) Adaptive optimal safety tracking control for multiplayer mixed zero-sum games of continuous-time systems. Appl Intell 1–16

  24. Shi L, Wang X, Cheng Y (2023) Safe reinforcement learning-based robust approximate optimal control for hypersonic flight vehicles. IEEE Transactions on vehicular technology

  25. Qin C, Qiao X, Wang J, Zhang D, Hou Y, Hu, S (2023) Barrier-Critic adaptive robust control of nonzero-sum differential games for uncertain nonlinear systems with state constraints. IEEE Transactions on Systems, Man, and Cybernetics: Systems

  26. Zhang Y, Zhao B, Liu D, Zhang S (2022) Adaptive dynamic programming-based event-triggered robust control for multiplayer nonzero-sum games with unknown dynamics1-4mmplease verify and confirm the term “multi-player” has been changed to “multiplayer” in the title of this article. IEEE Transactions on Cybernetics

  27. Zhao J, Na J, Gao G (2022) Robust tracking control of uncertain nonlinear systems with adaptive dynamic programming. Neurocomputing 471:21–30

    Article  Google Scholar 

  28. Yang X, Xu M, Wei Q (2023) Adaptive dynamic programming for nonlinear-constrained h\(\infty \) control. IEEE Transactions on Systems, Man, and Cybernetics: Systems

  29. Ji R, Ge SS, Li D (2023) Saturation-tolerant prescribed control for nonlinear systems with unknown control directions and external disturbances. IEEE Transactions on Cybernetics

  30. Xu S, He B (2023) Robust adaptive fuzzy fault tolerant control of robot manipulators with unknown parameters. IEEE Transactions on fuzzy systems

  31. Yang M, Ma H, Li X, Shang C, Shen Q (2022) Bus bridging for rail disruptions: a distributionally robust fuzzy optimization approach. IEEE Transactions on Fuzzy Systems

  32. Gutierrez-Oribio D, Orlov Y, Stefanou I, Plestan F (2022) Robust tracking for the diffusion equation using sliding-mode boundary control. In: 2022 IEEE 61st Conference on decision and control (CDC). IEEE, pp 6076–6081

  33. Chen J, Lyu L, Fei Z, Xia W, Sun X-M (2023) Event-triggered adaptive robust control for a class of uncertain nonlinear systems with application to mechatronic system. IEEE Transactions on Industrial Informatics

  34. Sun N, Liang D, Wu Y, Chen Y, Qin Y, Fang Y (2019) Adaptive control for pneumatic artificial muscle systems with parametric uncertainties and unidirectional input constraints. IEEE Trans Industr Inf 16(2):969–979

    Article  Google Scholar 

  35. Zhu Y, Zhao D, He H, Ji J (2016) Event-triggered optimal control for partially unknown constrained-input systems via adaptive dynamic programming. IEEE Trans Industr Electron 64(5):4101–4109

    Article  Google Scholar 

  36. Wu Q, Zhao B, Liu D, Polycarpou MM (2023) Event-triggered adaptive dynamic programming for decentralized tracking control of input constrained unknown nonlinear interconnected systems. Neural Netw 157:336–349

    Article  Google Scholar 

  37. Xue S, Luo B, Liu D, Li Y (2020) Adaptive dynamic programming based event-triggered control for unknown continuous-time nonlinear systems with input constraints. Neurocomputing 396:191–200

    Article  Google Scholar 

  38. Yang X, Zhao B (2020) Optimal neuro-control strategy for nonlinear systems with asymmetric input constraints. IEEE/CAA Journal of Automatica Sinica 7(2):575–583

    Article  MathSciNet  Google Scholar 

  39. Kong L, He W, Dong Y, Cheng L, Yang C, Li Z (2019) Asymmetric bounded neural control for an uncertain robot by state feedback and output feedback. IEEE Transactions on Systems, Man, and Cybernetics: Systems 51(3):1735–1746

    Google Scholar 

  40. Zhao Y, Wang H, Xu N, Zong G, Zhao X (2023) Reinforcement learning-based decentralized fault tolerant control for constrained interconnected nonlinear systems. Chaos, Solitons & Fractals 167:113034

  41. Qiao J, Li M, Wang D (2022) Asymmetric constrained optimal tracking control with critic learning of nonlinear multiplayer zero-sum games. IEEE Transactions on Neural Networks and Learning Systems

  42. Yang X, Wei Q (2020) Adaptive critic learning for constrained optimal event-triggered control with discounted cost. IEEE Transactions on Neural Networks and Learning Systems 32(1):91–104

    Article  Google Scholar 

  43. Sun Y, Li C, Qin H, Deng Z, Chen Z (2022) Robust neural network-based tracking control for unmanned surface vessels under deferred asymmetric constraints. Int J Robust Nonlinear Control 32(5):2741–2759

    Article  MathSciNet  Google Scholar 

  44. Yang X, Zhou Y, Gao Z (2023) Reinforcement learning for robust stabilization of nonlinear systems with asymmetric saturating actuators. Neural Netw 158:132–141

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant (62001359), in part by the Fundamental Research Funds for the Central Universities (ZYTS23163), in part by the Science and Technology Research Project of the Henan Province under Grant (222102240014, 232102211059, 232102211047).

Author information

Authors and Affiliations

  1. School of Artificial Intelligence, Henan University, Zhengzhou, 450000, China

    Dehua Zhang, Yuchen Wang & Kaijun Jiang

  2. School of Cyber Engineering, Xidian University, Xi’an, 710126, China

    Linlin Liang

Authors
  1. Dehua Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuchen Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kaijun Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Linlin Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Linlin Liang.

Ethics declarations

Conflicts of Interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, D., Wang, Y., Jiang, K. et al. Safe optimal robust control of nonlinear systems with asymmetric input constraints using reinforcement learning. Appl Intell 54, 1–13 (2024). https://doi.org/10.1007/s10489-023-05184-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-05184-1

Keywords

Search

Navigation