Investigating the multi-objective optimization of quality and efficiency using deep reinforcement learning | Applied Intelligence Skip to main content

Advertisement

Springer Nature Link
Log in

Investigating the multi-objective optimization of quality and efficiency using deep reinforcement learning

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

This study proposes a pipeline, which is based on deep reinforcement learning, aims to solve the multi-objective problem (MOP) on efficiency and quality in manufacturing. The rapid development in the area of artificial intelligent has caused a series of reactions that stirred the traditional manufacturing, pushing for the better machining quality and higher productivity. Despite all this, there has been very little research applying reinforcement learning to solve practical problems in milling process. The proposed pipeline is a two-step algorithm and makes full use of double deep Q network (DDQN) to settle the MOP of milling parameters. Firstly, surface roughness (Ra) and material removal rate (MRR) are selected as quality and efficiency indicators, respectively. In specific, the reliable prediction model of Ra is constructed on a small batch raw data via DDQN improved support vector regression (DDQN-SVR) rather than sophisticated and complex physical modeling. The MRR model is constructed by an accepted empirical formula. Then, DDQN is employed again to solve the MOP of satisfying minimum Ra and maximum MRR and compared to other accepted algorithms. Eventually, the optimal combination of machining parameters determined by entropy method was validated by experiment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Attar H, Ehtemam-Haghighi S, Kent D, Dargusch MS (2018) Recent developments and opportunities in additive manufacturing of titanium-based matrix composites: a review. Int J Mach Tools Manuf 133:85–102

    Article  Google Scholar 

  2. Lu X, Zhang H, Jia Z, Feng Y, Liang SY (2018) Cutting parameters optimization for MRR under the constraints of surface roughness and cutter breakage in micro-milling process. J Mech Sci Technol 32:3379–3388

    Article  Google Scholar 

  3. Wu D, Wang H, Zhang K, Zhao B, Lin X (2020) Research on adaptive CNC machining arithmetic and process for near-net-shaped jet engine blade. J Intell Manuf 31:717–744

    Article  Google Scholar 

  4. Zhu L, Li H, Yang J, Wang WS (2012) Research on theoretical modeling of 3D chip of orthogonal turn-milling. Dongbei Daxue Xuebao/Journal of Northeastern University 33:111–115

    Google Scholar 

  5. Bakhtiari H, Karimi M, Rezazadeh S (2016) Modeling, analysis and multi-objective optimization of twist extrusion process using predictive models and meta-heuristic approaches, based on finite element results. J Intell Manuf 27:463–473

    Article  Google Scholar 

  6. Lu J, Liao X, Li S, Ouyang H, Chen K, Huang B (2019) An effective ABC-SVM approach for surface roughness prediction in manufacturing processes. Complexity 2019:1–13

    Article  Google Scholar 

  7. Xiao Z, Liao X, Long Z, Li M (2017) Effect of cutting parameters on surface roughness using orthogonal array in hard turning of AISI 1045 steel with YT5 tool. Int J Adv Manuf Technol 93:273–282

    Article  Google Scholar 

  8. Tangjitsitcharoen S, Thesniyom P, Ratanakuakangwan S (2017) Prediction of surface roughness in ball-end milling process by utilizing dynamic cutting force ratio. J Intell Manuf 28:13–21

    Article  Google Scholar 

  9. Mumtaz J, Li Z, Imran M, Yue L, Jahanzaib M, Sarfraz S, Shehab E, Ismail SO, Afzal K (2019) Multi-objective optimisation for minimum quantity lubrication assisted milling process based on hybrid response surface methodology and multi-objective genetic algorithm. Adv Mech Eng 11

  10. Soepangkat B, Norcahyo R, Pramujati B, Wahid M (2019) Multi-objective optimization in face milling process with cryogenic cooling using grey fuzzy analysis and BPNN-GA methods, engineering computations, ahead-of-print

  11. Sugumaran V (2013) Developing Gaussian process model to predict the surface roughness in boring operation. International Journal of Engineering Trends and Technology 4:219–223

    Google Scholar 

  12. Zhang GJ, Li J, Chen Y, Huang Y, Shao XY, Li MZ (2014) Prediction of surface roughness in end face milling based on Gaussian process regression and cause analysis considering tool vibration. Int J Adv Manuf Tech 75:1357–1370

    Article  Google Scholar 

  13. Aich U, Banerjee S (2014) Modeling of EDM responses by support vector machine regression with parameters selected by particle swarm optimization. Appl Math Model 38:2800–2818

    Article  MATH  Google Scholar 

  14. Cao WD, Liu X, Ni JJ (2020) Parameter optimization of support vector regression using Henry gas solubility optimization algorithm. Ieee Access 8:88633–88642

    Article  Google Scholar 

  15. Lela B, Bajic D, Jozic S (2009) Regression analysis, support vector machines, and Bayesian neural network approaches to modeling surface roughness in face milling. Int J Adv Manuf Tech 42:1082–1088

    Article  Google Scholar 

  16. Zuperl U, Cus F (2012) System for off-line feedrate optimization and neural force control in end milling. International Journal of Adaptive Control and Signal Processing 26:105–123

    Article  MATH  Google Scholar 

  17. Saravanan R, Asokan P, Sachidanandam M (2002) A multi-objective genetic algorithm (GA) approach for optimization of surface grinding operations. Int J Mach Tools Manuf 42:1327–1334

    Article  Google Scholar 

  18. Warsi SS, Agha MH, Ahmad R, Jaffery SHI, Khan M (2019) Sustainable turning using multi-objective optimization: a study of Al 6061 T6 at high cutting speeds. Int J Adv Manuf Technol 100:843–855

    Article  Google Scholar 

  19. Zhang Y, Wang G-G, Li K, Yeh W-C, Jian M, Dong J (2020) Enhancing MOEA/D with information feedback models for large-scale many-objective optimization. Inf Sci 522:1–16

    Article  MathSciNet  MATH  Google Scholar 

  20. Cheng R, Jin Y, Olhofer M, Sendhoff B (2016) A reference vector guided evolutionary algorithm for many-objective optimization. IEEE T Evolut Comput 20:773–791

    Article  Google Scholar 

  21. Trivedi A, Srinivasan D, Sanyal K, Ghosh A (2016) A survey of multiobjective evolutionary algorithms based on decomposition. IEEE T Evolut Comput 21:440–462

    Google Scholar 

  22. Acherjee B, Maity D, Kuar A (2020) Optimization of correlated and conflicting responses of ECM process using flower pollination algorithm. International Journal of Applied Metaheuristic Computing 11:1–15

    Google Scholar 

  23. Ghosh T, Wang Y, Martinsen K, Wang K (2020) A surrogate-assisted optimization approach for multi-response end milling of aluminum alloy AA3105. Int J Adv Manuf Technol 111:2419–2439

    Article  Google Scholar 

  24. Naik S, Das SR, Dhupal D (2020) Experimental investigation, predictive modeling, parametric optimization and cost analysis in electrical discharge machining of Al-SiC metal matrix composite. Silicon 2020:1–24

    Google Scholar 

  25. Bhavsar SN, Aravindan S, Rao PV (2015) Investigating material removal rate and surface roughness using multi-objective optimization for focused ion beam (FIB) micro-milling of cemented carbide. Precis Eng-J Int Soc Precis Eng Nanotechnol 40:131–138

    Google Scholar 

  26. Ishibuchi H, Setoguchi Y, Masuda H, Nojima Y (2017) Performance of decomposition-based many-objective algorithms strongly depends on Pareto front shapes. IEEE T Evolut Comput 21:169–190

    Article  Google Scholar 

  27. Du W, Ding S (2021) A survey on multi-agent deep reinforcement learning: from the perspective of challenges and applications. Artif Intell Rev 54:3215–3238

    Article  Google Scholar 

  28. Levine S, Pastor P, Krizhevsky A, Ibarz J, Quillen D (2018) Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. Int J Robot Res 37:421–436

    Article  Google Scholar 

  29. Li X, Serlin Z, Yang G, Belta C (2019) A formal methods approach to interpretable reinforcement learning for robotic planning. Science Robotics 4:eaay6276

    Article  Google Scholar 

  30. Silver D, Huang A, Maddison CJ, Guez A, Sifre L, van den Driessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M, Dieleman S, Grewe D, Nham J, Kalchbrenner N, Sutskever I, Lillicrap T, Leach M, Kavukcuoglu K, Graepel T, Hassabis D (2016) Mastering the game of Go with deep neural networks and tree search. Nature 529:484−+

    Article  Google Scholar 

  31. Silver D, Schrittwieser J, Simonyan K, Antonoglou I, Huang A, Guez A, Hubert T, Baker L, Lai M, Bolton A, Chen Y, Lillicrap T, Hui F, Sifre L, Driessche G, Graepel T, Hassabis D (2017) Mastering the game of go without human knowledge. Nature 550:354–359

    Article  Google Scholar 

  32. Ding S, Zhao X, Xu X, Sun T, Jia W (2019) An effective asynchronous framework for small scale reinforcement learning problems. Appl Intell 49:4303–4318

    Article  Google Scholar 

  33. Li J, Monroe W, Ritter A (2016) D. Jurafsky. Deep Reinforcement Learning for Dialogue Generation

  34. B. Dhingra, L. Li, X. Li, J. Gao, Y.-N. Chen, F. Ahmed, L Deng, Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access, 2017

    Book  Google Scholar 

  35. Watkins JCH, Dayan P (1992) Q-learning. Mach Learn 8:279–292

    Article  MATH  Google Scholar 

  36. Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I, Wierstra D, Riedmiller M (2013) Playing Atari with deep reinforcement learning. Comput Sci 2013:1–9

    Google Scholar 

  37. Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G, Petersen S, Beattie C, Sadik A, Antonoglou I, King H, Kumaran D, Wierstra D, Legg S, Hassabis D (2015) Human-level control through deep reinforcement learning. Nature 518:529–533

    Article  Google Scholar 

  38. H. Van Hasselt, A. Guez, D. Silver, Deep Reinforcement Learning with Double Q-learning, (2015)

  39. Z. Wang, N. Freitas, M. Lanctot, Dueling network architectures for deep reinforcement learning, (2015) 1995–2003

    Google Scholar 

  40. Hasan MM, Lwin K, Imani M, Shabut A, Bittencourt LF, Hossain MA (2019) Dynamic multi-objective optimisation using deep reinforcement learning: benchmark, algorithm and an application to identify vulnerable zones based on water quality. Eng Appl Artif Intell 86:107–135

    Article  Google Scholar 

  41. Du W, Ding S, Zhang C, Du S (2021) Modified action decoder using Bayesian reasoning for multi-agent deep reinforcement learning. Int J Mach Learn Cybern 12:2947–2961

    Article  Google Scholar 

  42. Li K, Zhang T, Wang R (2020) Deep reinforcement learning for multi-objective optimization. IEEE Transactions on Cybernetics 2020:1–12

    Google Scholar 

  43. Lu R, Li Y-C, Li Y, Jiang J, Ding Y (2020) Multi-agent deep reinforcement learning based demand response for discrete manufacturing systems energy management. Appl Energy 276:115473

    Article  Google Scholar 

  44. W. Gang, Z. Mianhao, Optimization of cutting parameters in machining surface to reduce errors, 2011

    Book  Google Scholar 

  45. C.C. Chang, C.J. Lin, LIBSVM: a library for support vector machines, Acm T Intel Syst Tec, 2 (2011), LIBSVM

  46. Vapnik V (1995) The nature of statistical learning theory

  47. Brereton RG, Lloyd GR (2010) Support vector machines for classification and regression. Analyst 135:230–267

    Article  Google Scholar 

  48. Han F, Li L, Cai W, Li C, Deng X, Sutherland JW (2020) Parameters optimization considering the trade-off between cutting power and MRR based on linear decreasing particle swarm algorithm in milling. J Clean Prod 262:121388

    Article  Google Scholar 

  49. Moreira LC, Li WD, Lu X, Fitzpatrick ME (2019) Energy-efficient machining process analysis and optimisation based on BS EN24T alloy steel as case studies. Robot Comput Integr Manuf 58:1–12

    Article  Google Scholar 

  50. Cherkassky V, Ma YQ (2004) Practical selection of SVM parameters and noise estimation for SVM regression. Neural Netw 17:113–126

    Article  MATH  Google Scholar 

  51. Levis AA, Papageorgiou LG (2005) Customer demand forecasting via support vector regression analysis. Chem Eng Res Des 83:1009–1018

    Article  Google Scholar 

  52. Gupta AK, Guntuku SC, Desu RK, Balu A (2015) Optimisation of turning parameters by integrating genetic algorithm with support vector regression and artificial neural networks. Int J Adv Manuf Technol 77:331–339

    Article  Google Scholar 

  53. Deb K, Jain H (2014) An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, part I: solving problems with box constraints. IEEE T Evolut Comput 18:577–601

    Article  Google Scholar 

  54. Hou Y, Wu N, Li Z, Zhang Y, Qu T, Zhu Q (2020) Many-objective optimization for scheduling of crude oil operations based on NSGA-III with consideration of energy efficiency. Swarm and Evolutionary Computation 57:100714

    Article  Google Scholar 

  55. Lowe R, Wu Y, Tamar A, Harb J, Abbeel P (2017) I. Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments

  56. Lei W, Wen H, Wu J, Hou W (2021) MADDPG-based security situational awareness for smart grid with intelligent edge. Appl Sci 11:3101

    Article  Google Scholar 

  57. Behnamian J, Zandieh M, Ghomi SMTF (2010) A multi-phase covering Pareto-optimal front method to multi-objective parallel machine scheduling. Int J Prod Res 48:4949–4976

    Article  MATH  Google Scholar 

Download references

Acknowledgments

This research is supported by the National Natural Science Foundation of China (NSFC) (Grant No. 51665005 and 52165062), Natural Science Foundation of Guangxi Province (Grant No. 2020JJD160004 and 2019JJB160048), and Middle-aged and Young Teachers’ Basic Ability Promotion Project of Guangxi (Grant No. 2020KY10014).

Author information

Authors and Affiliations

  1. Guangxi Key Laboratory of Manufacturing Systems and Advance Manufacturing Technology, Guangxi University, Nanning, 535004, China

    Zhenhui Wang, Juan Lu, Chaoyi Chen, Junyan Ma & Xiaoping Liao

  2. Department of Mechanical and Marine Engineering, Beibu Gulf University, Qinzhou, 535011, China

    Juan Lu

Authors
  1. Zhenhui Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Juan Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chaoyi Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Junyan Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiaoping Liao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoping Liao.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Z., Lu, J., Chen, C. et al. Investigating the multi-objective optimization of quality and efficiency using deep reinforcement learning. Appl Intell 52, 12873–12887 (2022). https://doi.org/10.1007/s10489-022-03326-5

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03326-5

Keywords

Search

Navigation