Optimization of large-scale UAV cluster confrontation game based on integrated evolution strategy | Cluster Computing Skip to main content
Springer Nature Link
Log in

Optimization of large-scale UAV cluster confrontation game based on integrated evolution strategy

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

The development of large-scale cluster intelligence will inevitably lead to new problems of adversarial game control. Aiming at the problem of high dimension and high dynamics in the process of unmanned aerial vehicle (UAV) cluster confrontation game, and the traditional optimal control algorithm cannot meet the requirements of timeliness, the evolution strategies (ESs) optimization method is proposed and applied to large-scale UAV cluster. It effectively avoids the problem that it is difficult to obtain accurate gradients when using reinforcement learning to deal with high-dimensional models, and promotes autonomous UAVs to find strategies with higher performance. First, the confrontation game models including UAV motion, cluster behavior patterns and interaction are established. Second, two UAV cluster game algorithms using the OpenAI evolution strategy (OpenAI ES) and integrated evolution strategy (IES) are presented. Finally, the large-scale UAV attack and defense confrontation scenarios have been established, and different sampling proportions and different numbers of UAVs are fully simulated. The results show that the two proposed algorithms can effectively solve large-scale UAV cluster confrontation game problems, especially the adaptive IES algorithm, which has better performance and shows more strategic behavior for the UAVs, which improves the effectiveness and robustness of confrontation strategies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

Data will be made available on request.

References

  1. Howard, R.A. (1960). Dynamic programming and Markov processes.

  2. Peng, X.B., Coumans, E., Zhang, T., Lee, T.W., Tan, J., Levine, S. Learning agile robotic locomotion skills by imitating animals. arXiv preprint (arXiv: 2004.00784) (2020).

  3. Kiran, B.R., Sobh, I., Talpaert, V., Mannion, P., Al Sallab, A.A., Yogamani, S., Pérez, P.: Deep reinforcement learning for autonomous driving: A survey. IEEE Trans. Intell. Transp. Syst. (2021).

  4. Zhi-Hua, Z.: AlphaGo special session: an introduction. Acta Automatica Sinica 42(5), 670 (2016)

    Google Scholar 

  5. Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., Van Den Driessche, G., et al.: Mastering the game of Go with deep neural networks and tree search. Nature 529(7587), 484–489 (2016)

    Article  CAS  PubMed  ADS  Google Scholar 

  6. Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., et al.: Mastering the game of go without human knowledge. Nature 550(7676), 354–359 (2017)

    Article  CAS  PubMed  ADS  Google Scholar 

  7. Konda, V., Tsitsiklis, J.: Actor-critic algorithms. Adv. Neural Inf. Process. Syst. 12 (1999).

  8. Liu, J.T., He, M., Luo, L., et al.: Eigenvalue analysis of the pinning control system of unmanned aerial vehicle cluster. Syst. Eng. Electron. 44(2), 612–618 (2022)

    Google Scholar 

  9. Yang, Y., Wang, W., Liu, L., Dev, K., Qureshi, N.M.F.: AoI optimization in the UAV-aided traffic monitoring network under attack: a stackelberg game viewpoint. IEEE Trans. Intell. Transp. Syst. (2022). https://doi.org/10.1109/TITS.2022.315739

    Article  Google Scholar 

  10. Yang, Y., Wei, X., Xu, R., Peng, L.: Joint optimization of AoI, SINR, completeness, and energy in UAV-aided SDCNs: coalition formation game and cooperative order. IEEE Trans. Green Commun. Netw. 6(1), 265–280 (2021)

    Article  Google Scholar 

  11. Wang, W., Srivastava, G., Lin, J.C.W., Yang, Y., Alazab, M., Gadekallu, T.R.: Data freshness optimization under CAA in the UAV-aided MECN: a potential game perspective. IEEE Trans. Intell. Transp. Syst. (2022). https://doi.org/10.1109/TITS.2022.3167485

    Article  PubMed  Google Scholar 

  12. Li, X., Li, R.Q., Dong, H.Y.: Study on collective intelli- gence control based on model of cluster cooperative. Electr. Autom. 28(4), 3–5 (2006)

    Google Scholar 

  13. Liu, J.Y., Yue, S.H., Wang, G., et al.: Cooperative evolution algorithm of multi-agent system under complex tasks. Syst. Eng. Electron. 43(04), 991–1002 (2021)

    Google Scholar 

  14. Ramirez-Atencia, C., Camacho, D.: Handling cluster of UAVs based on evolutionary multi-objective optimization. Progress Artif. Intell. 6(3), 263–274 (2017)

    Article  Google Scholar 

  15. Fan, D.D., Theodorou, E., Reeder, J.: Evolving cost functions for model predictive control of multi-agent UAV combat clusters. In Proceedings of the genetic and evolutionary computation conference companion, pp. 55–56 (2017).

  16. Müller, N., Glasmachers, T.: Challenges in high-dimensional reinforcement learning with evolution strategies. In: Auger, A., Fonseca, C.M., Lourenço, N., Machado, P., Paquete, L., Whitley, D. (eds.) International conference on parallel problem solving from nature, pp. 411–423. Springer, Cham (2018)

    Chapter  Google Scholar 

  17. Hansen, N., Arnold, D.V., Auger, A.: Evolution strategies. In: Kacprzyk, J., Pedrycz, W. (eds.) Springer handbook of computational intelligence, pp. 871–898. Springer, Berlin, Heidelberg (2015)

    Chapter  Google Scholar 

  18. Li, Z., Lin, X., Zhang, Q., Liu, H.: Evolution strategies for continuous optimization: a survey of the state-of-the-art. Cluster Evol. Comput 56, 100694 (2020)

    Google Scholar 

  19. Plappert, M., Houthooft, R., Dhariwal, P., Sidor, S., Chen, R.Y., Chen, X., et al.: Parameter space noise for exploration. arXiv preprint (arXiv: 1706.01905) (2017).

  20. Mannor, S., Rubinstein, R.Y., Gat, Y.: The cross entropy method for fast policy search. In Proceedings of the 20th international conference on machine learning (ICML-03), pp. 512–519 (2003).

  21. De Boer, P.T., Kroese, D.P., Mannor, S., Rubinstein, R.Y.: A tutorial on the cross-entropy method. Ann. Oper. Res. 134(1), 19–67 (2005)

    Article  MathSciNet  Google Scholar 

  22. Sloss, A.N., Gustafson, S.: 2019 Evolutionary algorithms review. arXiv preprint (arXiv: 1906.08870) (2019).

  23. Mirjalili, S.: Evolutionary algorithms and neural networks: studies in computational intelligence, p. 780. Springer, Cham (2019)

    Book  Google Scholar 

  24. Salimans, T., Ho, J., Chen, X., Sidor, S., Sutskever, I.: Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint (arXiv: 1703.03864) (2017).

  25. Hansen, N., Ostermeier, A.: Completely derandomized self-adaptation in evolution strategies. Evol. Comput. 9(2), 159–195 (2001)

    Article  CAS  PubMed  Google Scholar 

  26. Hansen, N.: The CMA evolution strategy: a tutorial. arXiv preprint (arXiv: 1604.00772) (2016).

  27. Liu, G., Tang, J., Liu, C., Li, W.: Survey of cooperative behavior modeling technology for unmanned aerial vehicles clusters. Syst. Eng. Electron. 43(8), 2221–2231 (2021)

    Google Scholar 

  28. Wu, S.T., Fei, Y.H.: Flight control system. Beijing University of Aeronautics and Astronautics Press, Beijing (2005)

    Google Scholar 

  29. Padfield, G.D.: Helicopter flight dynamics: including a treatment of tiltrotor aircraft. John Wiley & Sons (2018)

    Book  Google Scholar 

  30. Song, B.W., Zhang, B.S., Jiang, J., Du, X.X., Wang, D.Z.: Estimation of equation of motion of four-rotor dish-shaped AUV and simulation research on its hydrodynamic characteristics. Acta Armamentarii 37(2), 299–306 (2016)

    Google Scholar 

  31. Reynolds, C.W.: Flocks, herds and schools: A distributed behavioral model. In Proceedings of the 14th annual conference on computer graphics and interactive techniques, pp. 25–34 (1987).

  32. Huning, A. (1976). Evolutionsstrategie. optimierung technischer systeme nach prinzipien der biologischen evolution (1976).

  33. Schwefel, H.P.: Evolutionsstrategien für die numerische optimierung. In: Schwefel, H.-P. (ed.) Numerische Optimierung von Computer-Modellen mittels der Evolutionsstrategie, pp. 123–176. Birkhäuser, Basel (1977)

    Chapter  Google Scholar 

  34. Wierstra, D., Schaul, T., Glasmachers, T., Sun, Y., Peters, J., Schmidhuber, J.: Natural evolution strategies. J. Mach. Learn. Res. 15(1), 949–980 (2014)

    MathSciNet  Google Scholar 

  35. Majid, A.Y., Saaybi, S., van Rietbergen, T., Francois-Lavet, V., Prasad, R.V., Verhoeven, C.: Deep reinforcement learning versus evolution strategies: a comparative survey. arXiv preprint (arXiv: 2110.01411) (2021).

  36. Tran, T.H., Sanza, C., Duthen, Y.: Evolving prediction weights using evolution strategy. In Proceedings of the 10th annual conference companion on genetic and evolutionary computation, pp. 2009–2016 (2008, July).

  37. Wierstra, D., Schaul, T., Glasmachers, T., Sun, Y., Schmidhuber, J.: Natural evolution strategies. arXiv preprint (arXiv: 1106.4487) (2011).

  38. Sun, Y., Wierstra, D., Schaul, T., Schmidhuber, J.: Efficient natural evolution strategies. In Proceedings of the 11th annual conference on genetic and evolutionary computation, pp. 539–546 (2009).

  39. Sun, Y., Wierstra, D., Schaul, T., Schmidhuber, J.: Stochastic search using the natural gradient. In Proceedings of the 26th annual international conference on machine learning, pp. 1161–1168 (2009, June).

  40. Ruder, S.: An overview of gradient descent optimization algorithms. arXiv preprint (arXiv: 1609.04747) (2016).

  41. Yao, X., Liu, Y.: Fast evolution strategies. In: Angeline, P.J., Reynolds, R.G., McDonnell, J.R., Eberhart, R. (eds.) International conference on evolutionary programming, pp. 149–161. Springer, Berlin, Heidelberg (1997)

    Google Scholar 

  42. Lee, C.Y., Park, Y.J.: Effect of multivariate Cauchy mutation in evolutionary programming. IEICE Trans. Inf. Syst. 97(4), 821–829 (2014)

    Article  ADS  Google Scholar 

  43. Mallipeddi, R., Mallipeddi, S., Suganthan, P.N.: Ensemble strategies with adaptive evolutionary programming. Inf. Sci. 180(9), 1571–1581 (2010)

    Article  Google Scholar 

  44. Yao, X., Liu, Y., Lin, G.: Evolutionary programming made faster. IEEE Trans. Evol. Comput. 3(2), 82–102 (1999)

    Article  Google Scholar 

  45. Choi, T.J., Ahn, C.W., An, J.: An adaptive Cauchy differential evolution algorithm for global numerical optimization. Sci. World J. 2013, 1–12 (2013)

    Google Scholar 

  46. Ajani, O.S., Mallipeddi, R.: Adaptive evolution strategy with ensemble of mutations for reinforcement learning. Knowl.-Based Syst. 245, 108624 (2022)

    Article  Google Scholar 

  47. Xue, N., Niu, L., Hong, X., Li, Z., Hoffaeller, L., Pöpper, C.: Deepsim: Gps spoofing detection on uavs using satellite imagery matching. In Annual computer security applications conference, pp. 304–319 (2020).

Download references

Acknowledgements

The authors also gratefully acknowledge the helpful comments and suggestions of the reviewers.

Funding

This work was supported in part by the equipment advance research project (50912020401), the aviation science foundation of China (201908052002) and the Hunan Key Laboratory of intelligent decision-making technology for emergency management (2020TP1013).

Author information

Authors and Affiliations

  1. College of Astronautics, Nanjing University of Aeronautics and Astronautics, Nanjing, 210016, China

    Haiying Liu & Kun Wu

  2. College of System Engineering, National University of Defense Technology, Changsha, 410073, China

    Kuihua Huang, Guangquan Cheng & Rui Wang

  3. College of Mathematics, Southeast University, Nanjing, 210096, China

    Guohua Liu

  4. Nanjing Center for Applied Mathematics, Nanjing, 211135, China

    Haiying Liu

Authors
  1. Haiying Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Kun Wu
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Kuihua Huang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Guangquan Cheng
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Rui Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Guohua Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by HL, KW and KH. The first draft of the manuscript was written by HL and KW, and all authors commented on previous versions of the manuscript.

Corresponding author

Correspondence to Haiying Liu.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, H., Wu, K., Huang, K. et al. Optimization of large-scale UAV cluster confrontation game based on integrated evolution strategy. Cluster Comput 27, 515–529 (2024). https://doi.org/10.1007/s10586-022-03961-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-022-03961-0

Keywords