A Dynamic Resource Allocation Strategy with Reinforcement Learning for Multimodal Multi-objective Optimization | Machine Intelligence Research Skip to main content
Springer Nature Link
Log in

A Dynamic Resource Allocation Strategy with Reinforcement Learning for Multimodal Multi-objective Optimization

  • Research Article
  • Published:
Machine Intelligence Research Aims and scope Submit manuscript

Abstract

Many isolation approaches, such as zoning search, have been proposed to preserve the diversity in the decision space of multimodal multi-objective optimization (MMO). However, these approaches allocate the same computing resources for subspaces with different difficulties and evolution states. In order to solve this issue, this paper proposes a dynamic resource allocation strategy (DRAS) with reinforcement learning for multimodal multi-objective optimization problems (MMOPs). In DRAS, relative contribution and improvement are utilized to define the aptitude of subspaces, which can capture the potentials of subspaces accurately. Moreover, the reinforcement learning method is used to dynamically allocate computing resources for each subspace. In addition, the proposed DRAS is applied to zoning searches. Experimental results demonstrate that DRAS can effectively assist zoning search in finding more and better distributed equivalent Pareto optimal solutions in the decision space.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. S. L. Karri, L. C. De Silva, D. T. C. Lai, S. Y. Yong. Identification and classification of driving behaviour at signalized intersections using support vector machine. International Journal of Automation and Computing, vol. 18, no. 3, pp. 480–491, 2021. DOI: https://doi.org/10.1007/s11633-021-1295-y.

    Article  Google Scholar 

  2. H. T. Ye, Z. Q. Li. PID neural network decoupling control based on hybrid particle swarm optimization and differential evolution. International Journal of Automation and Computing, vol. 17, no. 6, pp. 867–872, 2020. DOI: https://doi.org/10.1007/s11633-015-0917-7.

    Article  Google Scholar 

  3. W. Jia, W. Xia, Y. Zhao, H. Min, Y. X. Chen. 2D and 3D palmprint and palm vein recognition based on neural architecture search. International Journal of Automation and Computing, vol. 18, no. 3, pp. 377–409, 2021. DOI: https://doi.org/10.1007/s11633-021-1292-1.

    Article  Google Scholar 

  4. A. Jaszkiewicz. On the performance of multiple-objective genetic local search on the 0/1 knapsack problem-a comparative experiment. IEEE Transactions on Evolutionary Computation, vol. 6, no. 4, pp. 402–412, 2002. DOI: https://doi.org/10.1109/TEVC.2002.802873.

    Article  Google Scholar 

  5. K. Deb. Multi-objective genetic algorithms: Problem difficulties and construction of test problems. Evolutionary Computation, vol. 7, no. 3, pp. 205–230, 1999. DOI: https://doi.org/10.1162/evco.1999.7.3.205.

    Article  Google Scholar 

  6. Q. F. Zhang, H. Li. MOEA/D: A multiobjective evolutionary algorithm based on decomposition. IEEE Transactions on Evolutionary Computation, vol. 11, no. 6, pp. 712–731, 2007. DOI: https://doi.org/10.1109/TEVC.2007.892759.

    Article  Google Scholar 

  7. K. Deb, A. Pratap, S. Agarwal, T. Meyarivan. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation, vol. 6, no. 2, pp. 182–197, 2002. DOI: https://doi.org/10.1109/4235.996017.

    Article  Google Scholar 

  8. S. Lu, Y. M. Li, B. X. Ding. Multi-objective dimensional optimization of a 3-DOF translational PKM considering transmission properties. International Journal of Automation and Computing, vol. 16, no. 6, pp. 748–760, 2019. DOI: https://doi.org/10.1007/s11633-019-1184-9.

    Article  Google Scholar 

  9. P. S. Oliveto, D. Sudholt, C. Zarges. On the benefits and risks of using fitness sharing for multimodal optimisation. Theoretical Computer Science, vol. 773, pp. 53–70, 2019. DOI: https://doi.org/10.1016/j.tcs.2018.07.007.

    Article  MathSciNet  MATH  Google Scholar 

  10. C. Y. Lin, W. H. Wu. Niche identification techniques in multimodal genetic search with sharing scheme. Advances in Engineering Software, vol. 33, no. 11–12, pp. 779–791, 2002. DOI: https://doi.org/10.1016/S0965-9978(02)00045-5.

    Article  Google Scholar 

  11. R. Thomsen. Multimodal optimization using crowding-based differential evolution. In Proceedings of Congress on Evolutionary Computation, IEEE, Portland, USA, pp. 1382–1389, 2004. DOI: https://doi.org/10.1109/CEC.2004.1331058.

    Google Scholar 

  12. M. Q. Li, D. Lin, J. S. Kou. A hybrid niching PSO enhanced with recombination-replacement crowding strategy for multimodal function optimization. Applied Soft Computing, vol. 12, no. 3, pp. 975–987, 2012. DOI: https://doi.org/10.1016/j.asoc.2011.11.032.

    Article  Google Scholar 

  13. J. P. Li, M. E. Balazs, G. T. Parks, P. J. Clarkson. A species conserving genetic algorithm for multimodal function optimization. Evolutionary Computation, vol. 10, no. 3, pp. 207–234, 2002. DOI: https://doi.org/10.1162/106365602760234081.

    Article  Google Scholar 

  14. Q. Q. Fan, X. F. Yan. Solving multimodal multiobjective problems through zoning search. IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 51, no. 8, pp. 4836–4847, 2021. DOI: https://doi.org/10.1109/TSMC.2019.2944338.

    Article  Google Scholar 

  15. K. Miettinen. Nonlinear Multiobjective Optimization, Boston, USA: Kluwer Academic Publishers, 1999.

    MATH  Google Scholar 

  16. J. J. Liang, C. T. Yue, B. Y. Qu. Multimodal multi-objective optimization: A preliminary study. In Proceedings of IEEE Congress on Evolutionary Computation, IEEE, Vancouver, Canada, pp. 2454–2461, 2016. DOI: https://doi.org/10.1109/CEC.2016.7744093.

    Google Scholar 

  17. S. C. Maree, T. Alderliesten, P. A. N. Bosman. Real-valued evolutionary multi-modal multi-objective optimization by hill-valley clustering. In Proceedings of the Genetic and Evolutionary Computation Conference, Association for Computing Machinery, Prague, Czech Republic, pp. 568–576, 2019. DOI: https://doi.org/10.1145/3321707.3321759.

    Google Scholar 

  18. C. T. Yue, B. Y. Qu, J. Liang. A multi-objective particle swarm optimizer using ring topology for solving multimodal multi-objective problems. IEEE Transactions on Evolutionary Computation, vol. 22, no. 5, pp. 805–817, 2018. DOI: https://doi.org/10.1109/TEVC.2017.2754271.

    Article  Google Scholar 

  19. R. Tanabe, H. Ishibuchi. A decomposition-based evolutionary algorithm for multi-modal multi-objective optimization. In Proceedings of the 15th International Conference on Parallel Problem Solving from Nature, Springer, Coimbra, Portugal, pp. 249–261, 2018. DOI: https://doi.org/10.1007/978-3-319-99253-2_20.

    Google Scholar 

  20. R. Tanabe, H. Ishibuchi. A framework to handle multimodal multiobjective optimization in decomposition-based evolutionary algorithms. IEEE Transactions on Evolutionary Computation, vol. 24, no. 4, pp. 720–734, 2020. DOI: https://doi.org/10.1109/TEVC.2019.2949841.

    Article  Google Scholar 

  21. Y. M. Peng, H. Ishibuchi. A decomposition-based large-scale multi-modal multi-objective optimization algorithm. In Proceedings of IEEE Congress on Evolutionary Computation, IEEE, Glasgow, UK, pp. 1–8, 2020. DOI: https://doi.org/10.1109/CEC48606.2020.9185674.

    Google Scholar 

  22. R. Tanabe, H. Ishibuchi. A niching indicator-based multimodal many-objective optimizer. Swarm and Evolutionary Computation, vol. 49, pp. 134–146, 2019. DOI: https://doi.org/10.1016/j.swevo.2019.06.001.

    Article  Google Scholar 

  23. Q. F. Zhang, W. D. Liu, H. Li. The performance of a new version of MOEA/D on CEC09 unconstrained MOP test instances. In Proceedings of IEEE Congress on Evolutionary Computation, IEEE, Trondheim, Norway, pp. 203–208, 2009. DOI: https://doi.org/10.1109/CEC.2009.4982949.

    Google Scholar 

  24. Q. Kang, X. Y. Song, M. C. Zhou, L. Li. A collaborative resource allocation strategy for decomposition-based multiobjective evolutionary algorithms. IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 49, no. 12, pp. 2416–2423, 2019. DOI: https://doi.org/10.1109/TSMC.2018.2818175.

    Article  Google Scholar 

  25. A. M. Zhou, Q. F. Zhang. Are all the subproblems equally important? Resource allocation in decomposition-baeed multiobjective evolutionary algorithms IEEE Transactions on Evolutionary Computation, vol. 20, no. 1, pp. 52–64, 2016. DOI: https://doi.org/10.1109/TEVC.2015.2424251.

    Article  Google Scholar 

  26. Q. Z. Lin, G. M. Jin, Y. P. Ma, K. C. Wong, C. A. Coello Coello, J. Q. Li, J. Y. Chen, J. Zhang. A diversrty-enhanced resource allocation strategy for decomposition-based multiobjective evolutionary algorithm. IEEE Transactions on Cybernetics, vol. 48, no. 8, pp. 2388–2401, 2018. DOI: https://doi.org/10.1109/TCYB.2017.2739185.

    Article  Google Scholar 

  27. X. Y. Caí, Y. X. Li, Z. Fan, Q. F. Zhang. An external archive guided multiobjective evolutionary algorithm based on decomposition for combinatorial optimization. IEEE Transactions on Evolutionary Computation, vol. 19, no. 4, pp. 508–523, 2015. DOI: https://doi.org/10.1109/TEVC.2014.2350995.

    Article  Google Scholar 

  28. Y. Xiang, Y. R. Zhou, L. P. Tang, Z. F. Chen. A decomposition-based many-objective artificial bee colony algorithm. IEEE Transactions on Cybernetics, vol. 49, no. 1, pp. 287–300, 2019. DOI: https://doi.org/10.1109/TCYB.2017.2772250.

    Article  Google Scholar 

  29. K. Li, A. Fialho, S. Kwong, Q. F. Zhang. Adaptive operator selection with bandits for a multiobjective evolutionary algorithm based on decomposition. IEEE Transactions on Evolutionary Computation, vol. 18, no. 1, pp. 114–130, 2014. DOI: https://doi.org/10.1109/TEVC.2013.2239648.

    Article  Google Scholar 

  30. H. L. Liu, L. Chen, Q. F. Zhang, K. Deb. Adaptively allocating search effort in challenging many-objective optimization problems. IEEE Transactions on Evolutionary Computation, vol. 22, no. 3, pp. 433–448, 2018. DOI: https://doi.org/10.1109/TEVC.2017.2725902.

    Article  Google Scholar 

  31. H. K. Chen, G. H. Wu, W. Pedrycz, P. N. Suganthan, L. N. Xing, X. M. Zhu. An adaptive resource allocation strategy for objective space partition-based multiobjective optimization. IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 51, no. 3, pp. 1507–1522, 2021. DOI: https://doi.org/10.1109/TSMC.2019.2898456.

    Google Scholar 

  32. J. J. Zhou, L. Gao, X. Y. Li. Ensemble of dynamic resource allocation strategies for decomposition-based multi-objective optimization. IEEE Transactions on Evolutionary Computation, vol. 25, no. 4, pp. 710–723, 2021. DOI: https://doi.org/10.1109/TEVC.2021.3060899.

    Article  Google Scholar 

  33. F. van den Bergh, A. P. Engelbrecht. A cooperative approach to particle swarm optimization. IEEE Transactions on Evolutionary Computation, vol. 8, no. 3, pp. 225–239, 2004. DOI: https://doi.org/10.1109/TEVC.2004.826069.

    Article  Google Scholar 

  34. Y. J. Shi, H. F. Teng, Z. Q. Li. Cooperative co-evolutionary differential evolution for function optimization. In Proceedings of the 1st International Conference on Advances in Natural Computation, Springer, Changsha, China, pp. 1080–1088, 2005. DOI: https://doi.org/10.1007/11539117_147.

    Google Scholar 

  35. J. J. Liang, A. K. Qin, P. N. Suganthan, S. Baskar. Comprehensive learning particle swarm optimizer for global optimization of multimodal functions. IEEE Transactions on Evolutionary Computation, vol. 10, no. 3, pp. 281–295, 2006. DOI: https://doi.org/10.1109/TEVC.2005.857610.

    Article  Google Scholar 

  36. Z. Y. Yang, K. Tang, X. Yao. Large scale evolutionary optimization using cooperative coevolution. Information Sciences, vol. 178, no. 15, pp. 2985–2999, 2008. DOI: https://doi.org/10.1016/j.ins.2008.02.017.

    Article  MathSciNet  MATH  Google Scholar 

  37. M. Yang, M. N. Omidvar, C. H. Li, X. D. Li, Z. H. Cai, B. Kazimipour, X. Yao. Efficient resource allocation in cooperative co-evolution for large-scale global optimization. IEEE Transactions on Evolutionary Computation, vol. 21, no. 4, pp. 493–505, 2017. DOI: https://doi.org/10.1109/TEVC.2016.2627581.

    Article  Google Scholar 

  38. Y. H. Jia, W. N. Chen, T. L. Gu, H. X. Zhang, H. Q. Yuan, S. Kwong, J. Zhang. Distributed cooperative co-evolution with adaptive computing resource allocation for large scale optimization. IEEE Transactions on Evolutionary Computation, vol. 23, no. 2, pp. 188–202, 2019. DOI: https://doi.org/10.1109/TEVC.2018.2817889.

    Article  Google Scholar 

  39. X. N. Shen, Y. N. Guo, A. M. Li. Cooperative coevolution with an improved resource allocation for large-scale multi-objective software project scheduling. Applied Soft Computing, vol. 88, Article number 106059, 2020. DOI: https://doi.org/10.1016/j.asoc.2019.106059.

  40. Y. H. Jia, Y. Mei, M. J. Zhang. Contribution-based cooperative co-evolution for nonseparable large-scale problems with overlapping subcomponents. IEEE Transactions on Cybernetics, to be published. DOI: https://doi.org/10.1109/TCYB.2020.3025577.

  41. M. G. Gong, Z. D. Tang, H. Li, J. Zhang. Evolutionary multitasking with dynamic resource allocating strategy. IEEE Transactions on Evolutionary Computation, vol. 23, no. 5, pp. 858–869, 2019. DOI: https://doi.org/10.1109/TEVC.2019.2893614.

    Article  Google Scholar 

  42. Y. Wang, H. C. Tan, Y. K. Wu, J. K. Peng. Hybrid electric vehicle energy management with computer vision and deep reinforcement learning. IEEE Transactions on Industrial Informatics, vol. 17, no. 6, pp. 3857–3868, 2021. DOI: https://doi.org/10.1109/TII.2020.3015748.

    Article  Google Scholar 

  43. Y. Deng, F. Bao, Y. Y. Kong, Z. Q. Ren, Q. H. Dai. Deep direct reinforcement learning for financial signal representation and trading. IEEE Transactions on Neural Networks and Learning Systems, vol. 28, no. 3, pp. 653–664, 2017. DOI: https://doi.org/10.1109/TNNLS.2016.2522401.

    Article  Google Scholar 

  44. Y. X. Wang, K. Wang, H. W. Huang, T. M. Miyazaki, S. Guo. Traffic and computation co-offloading with reinforcement learning in fog computing for industrial applications. IEEE Transactions on Industrial Informatics, vol. 15, no. 2, pp. 976–986, 2019. DOI: https://doi.org/10.1109/TII.2018.2883991.

    Article  Google Scholar 

  45. W. Y. Wang, J. W. Li, X. D. He. Deep reinforcement learning for NLP. in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, Association for Computational Linguistics, Melbourne, Australia, pp. 19–21, 2018. DOI: https://doi.org/10.18653/v1/P18-5007.

    Google Scholar 

  46. Z. Q. Wan, C. Jiang, M. Fahad, Z. Ni, Y. Guo, H. B. He. Robot-assisted pedestrian regulation based on deep reinforcement learning. IEEE Transactions on Cybernetics, vol. 50, no. 4, pp. 1669–1682, 2020. DOI: https://doi.org/10.1109/TCYB.2018.2878977.

    Article  Google Scholar 

  47. E. Mocanu, D. C. Mocanu, P. H. Nguyen, A. Liotta, M. E. Webber, M. Gibescu, J. G. Slootweg. On-line building energy optimization using deep reinforcement learning. IEEE Transactions on Smart Grid, vol. 10, no. 4, pp. 3698–3708, 2019. DOI: https://doi.org/10.1109/TSG.2018.2834219.

    Article  Google Scholar 

  48. X. Y. Zhang, Y. Tian, Y. C. Jin. A knee point-driven evolutionary algorithm for many-objective optimization. IEEE Transactions on Evolutionary Computation, vol. 19, no. 6, pp. 761–776, 2015. DOI: https://doi.org/10.1109/TEVC.2014.2378512.

    Article  Google Scholar 

  49. K. X. Wei, A. Aviles-Rivero, J. W. Liang, Y. Fu, C. B. Schonlieb, H. Huang. Tuning-free plug-and-play proximal algorithm for inverse imaging problems. In Proceedings of the 37th International Conference on Machine Learning, ICML, Vienna, Austria, pp. 10158–10169, 2020.

    Google Scholar 

  50. J. Liang, B. Y. Qu, D. W. Gong, C. T. Yue. Problem definitions and evaluation criteria for the CEC 2019 special session on multimodal multiobjective optimization. Technical Report, Zhengzhou University, China, 2019. DOI: https://doi.org/10.1314/2.2.33423.64164.

    Google Scholar 

  51. E. Zitzler, L. Thiele. Multiobjective evolutionary algorithms: A comparative case study and the strength Pareto approach. IEEE Transactions on Evolutionary Computation, vol. 3, no. 4, pp. 257–271, 1999. DOI: https://doi.org/10.1109/4235.797969.

    Article  Google Scholar 

  52. S. Bandyopadhyay, S. K. Pal, B. Aruna. Multiobjective GAs, quantitative indices, and pattern classification. IEEE Transactions on Systems, Man, and Cybernetics - Part B: Cybernetics, vol. 34, no. 5, pp. 2088–2099, 2004. DOI: https://doi.org/10.1109/TSMCB.2004.834438.

    Article  Google Scholar 

  53. B. Y. Qu, C. Li, J. Liang, L. Yan, K. J. Yu, Y. S. Zhu. A self-organized speciation based multi-objective particle swarm optimizer for multimodal multi-objective problems. Applied Soft Computing, vol. 86, Article number 105886, 2020. DOI: https://doi.org/10.1016/j.asoc.2019.105886.

  54. M. Friedman. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association, vol. 32, no. 200, pp. 675–701, 1937. DOI: https://doi.org/10.1080/01621459.1937.10503522.

    Article  MATH  Google Scholar 

  55. Y. Liu, H. Ishibuchi, G. G. Yen, Y. Nojima, N. Masuyama. Handling imbalance between convergence and diversity in the decision space in evolutionary multi-modal multi-objective optimization. IEEE Transactions on Evolutionary Computation, vol. 24, no. 3, pp. 551–565, 2020. DOI: https://doi.org/10.1109/TEVC.2019.2938557.

    Google Scholar 

  56. Y. P. Liu, H. Ishibuchi, Y. Nojima, N. Masuyama, K. Shang. A double-niched evolutionary algorithm and its behavior on polygon-based problems. In Proceedings of the 15th International Conference on Parallel Problem Solving from Nature, Springer, Coimbra, Portugal, pp. 262–273, 2018. DOI: https://doi.org/10.1007/978-3-319-99253-2_21.

    Google Scholar 

  57. Y. P. Liu, G. G. Yen, D. W. Gong. A multimodal multiobjective evolutionary algorithm using two-archive and recombination strategies. IEEE Transactions on Evolutionary Computation, vol. 23, no. 4, pp. 660–674, 2019. DOI: https://doi.org/10.1109/TEVC.2018.2879406.

    Article  Google Scholar 

  58. K. Deb, S. Tiwari. Omni-optimizer: A generic evolutionary algorithm for single and multi-objective optimization. European Journal of Operational Research, vol. 185, no. 3, pp. 1062–1087, 2008. DOI: https://doi.org/10.1016/j.ejor.2006.06.042.

    Article  MathSciNet  MATH  Google Scholar 

  59. A. M. Zhou, Q. F. Zhang, Y. C. Jin. Approximating the set of Pareto-optimal solutions in both the decision and objective spaces by an estimation of distribution algorithm. IEEE Transactions on Evolutionary Computation, vol. 13, no. 5, pp. 1167–1189, 2009. DOI: https://doi.org/10.1109/TEVC.2009.2021467.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Mathematics and Statistics, Xidian University, Xi’an, 710126, China

    Qian-Long Dang, Wei Xu & Yang-Fei Yuan

Authors
  1. Qian-Long Dang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wei Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yang-Fei Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qian-Long Dang.

Additional information

Colored figures are available in the online version at https://link.springer.com/journal/11633

Qian-Long Dang received the B. Eng. degree in applied mathematics from Henan University of Technology, China in 2016. He is currently a Ph. D. degree candidate in School of Mathematics and Statistics, Xidian University, China.

His research interests include computational intelligence, swarm intelligence, evolution algorithm, and their applications.

Wei Xu received the B. Eng. degree in applied mathematics from Inner Mongolia University of Science and Technology, China in 2018. He is currently a master student in School of Mathematics and Statistics, Xidian University, China.

His research interests include computational intelligence, swarm intelligence, evolution algorithm, and their applications.

Yang-Fei Yuan received the B. Eng. degree in applied mathematics from Inner Mongolia University of Science and Technology, China in 2018. He is currently a master student in School of Mathematics and Statistics, Xidian University, China.

His research interests include computational intelligence, swarm intelligence, evolution algorithm, and their applications.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dang, QL., Xu, W. & Yuan, YF. A Dynamic Resource Allocation Strategy with Reinforcement Learning for Multimodal Multi-objective Optimization. Mach. Intell. Res. 19, 138–152 (2022). https://doi.org/10.1007/s11633-022-1314-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11633-022-1314-7

Keywords

Search

Navigation