Path planning of modular robots on various terrains using Q-learning versus optimization algorithms | Intelligent Service Robotics Skip to main content
Springer Nature Link
Log in

Path planning of modular robots on various terrains using Q-learning versus optimization algorithms

  • Original Research Paper
  • Published:
Intelligent Service Robotics Aims and scope Submit manuscript

Abstract

Self-reconfigurable modular robots (SRMRs) have recently attracted considerable attention because of their numerous potential applications in the real world. In this paper, we draw a comprehensive comparison among five different algorithms in path planning of a novel SRMR system called ACMoD through an environment comprised of various terrains in a static condition. The contribution of this work is that the reconfiguration ability of ACMoD has been taken into account. This consideration, though raises new algorithmic challenges, equips the robot with new capability to pass difficult terrains rather than bypassing them, and consequently the robot can achieve better performance in terms of traversal time and energy consumption. In this work, four different optimization algorithms, including Adaptive Genetic Algorithm, Elitist Ant System, Dijkstra and Dynamic Weighting A*, along with a well-known reinforcement learning algorithm called Q-Learning, are proposed to solve this path planning problem. The outputs of these algorithms are the optimal path through the environment and the associated configuration on each segment of the path. The challenges involved in mapping the path planning problem to each algorithm are discussed in full details. Eventually, all algorithms are compared in terms of the quality of their solutions and convergence rate.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Bhat P et al (2006) Hierarchical motion planning for self-reconfigurable modular robots. In: 2006 IEEE/RSJ on international conference on intelligent robots and systems

  2. Liu T et al (2009) The adaptive path planning research for a shape-shifting robot using particle swarm optimization. In: Fifth international conference on natural computation, 2009

  3. Sun X et al (2015) A reconfiguration approach for self-reconfigurable modular robot using assisted modules. In: 2015 IEEE international conference on mechatronics and automation (ICMA)

  4. Christensen D et al (2014) Collective modular underwater robotic system for long-term autonomous operation. Science 19(1):35–40

    MathSciNet  Google Scholar 

  5. Hancher MD, Hornby GS (2006) A modular robotic system with applications to space exploration. In: 2nd IEEE international conference on space mission challenges for information technology (SMC-IT’06)

  6. Liu S et al (2012) A reconfigurable modular robot for detection and rescue. In: Lei J, Wang FL, Deng H, Miao D (eds) Emerging research in artificial intelligence and computational intelligence. Springer, 17–24

  7. Tsai C-C, Huang H-C, Chan C-K (2011) Parallel elite genetic algorithm and its application to global path planning for autonomous robot navigation. IEEE Trans Ind Electron 58(10):4813–4821

    Article  Google Scholar 

  8. Soltani AR et al (2002) Path planning in construction sites: performance evaluation of the Dijkstra, A*, and GA search algorithms. Adv Eng Inform 16(4):291–303

    Article  Google Scholar 

  9. Naderan-Tahan M, Manzuri-Shalmani MT (2009) Efficient and safe path planning for a mobile robot using genetic algorithm. In: IEEE congress on evolutionary computation, 2009. CEC’09

  10. Guanghua Z, Zhicheng D, Wei W (2006) Realization of a modular reconfigurable robot for rough terrain. In: Proceedings of the 2006 IEEE international conference on mechatronics and automation

  11. Golestan K, Asadpour M, Moradi H (2013) A new graph signature calculation method based on power centrality for modular robots. In: Martinoli A, Mondada F, Correll N, Mermoud G, Egerstedt M, Hsieh MA, Parker LE, St\(\varnothing \)y K (eds) Distributed autonomous robotic systems. Springer, pp 505–516

  12. Yim M et al (2007) Modular self-reconfigurable robot systems [grand challenges of robotics]. IEEE Robot Autom Mag 14(1):43–52

    Article  Google Scholar 

  13. Rus D, Vona M (2001) Crystalline robots: self-reconfiguration with compressible unit modules. Auton Robots 10(1):107–124

    Article  MATH  Google Scholar 

  14. Jorgensen MW, Ostergaard EH, Lund HH (2004) Modular ATRON: modules for a self-reconfigurable robot. In: Proceedings of 2004 IEEE/RSJ international conference on intelligent robots and systems, 2004 (IROS 2004)

  15. Murata S et al (2002) M-TRAN: self-reconfigurable modular robotic system. IEEE/ASME Trans Mechatron 7(4):431–441

    Article  Google Scholar 

  16. Castano A, Shen W-M, Will P (2000) CONRO: towards deployable robots with inter-robots metamorphic capabilities. Auton Robots 8(3):309–324

    Article  Google Scholar 

  17. Ryland GG, Cheng HH (2010) Design of iMobot, an intelligent reconfigurable mobile robot with novel locomotion. In: 2010 IEEE international conference on robotics and automation (ICRA)

  18. Haghzad S, Bagheri S, Faraji S (2013) Finding proper configurations for modular robots by using genetic algorithm on different terrains. Int J Mater Mech Manuf 1:360–365

    Google Scholar 

  19. Fetanat M, Haghzad S, Shouraki SB (2015) Optimization of dynamic mobile robot path planning based on evolutionary methods. In: IEEE AI & Robotics (IRANOPEN), 2015

  20. Hu Y, Yang SX (2004) A knowledge based genetic algorithm for path planning of a mobile robot. In: Proceedings of 2004 IEEE international conference on robotics and automation, 2004. ICRA’04

  21. Konar A et al (2013) A deterministic improved Q-learning for path planning of a mobile robot. IEEE Trans Syst Man Cybern Syst 43(5):1141–1153

    Article  Google Scholar 

  22. Wang H, Yu Y, Yuan Q (2011) Application of Dijkstra algorithm in robot path-planning. In: 2011 second international conference on mechanic automation and control engineering

  23. Cai C, Ferrari S (2009) Information-driven sensor path planning by approximate cell decomposition. IEEE Trans Syst Man Cybern Part B Cybern 39(3):672–689

    Article  Google Scholar 

  24. Yu-qin W, Xiao-peng Y (2012) Research for the robot path planning control strategy based on the immune particle swarm optimization algorithm. In: 2012 second international conference on intelligent system design and engineering application (ISDEA)

  25. Dong H et al (2010) The path planning for mobile robot based on Voronoi diagram. In: 2010 3rd international conference on intelligent networks and intelligent systems (ICINIS)

  26. Janet JA, Luo RC, Kay MG (1995) The essential visibility graph: an approach to global motion planning for autonomous mobile robots. In: Proceedings of 1995 IEEE international conference on robotics and automation, 1995

  27. Xin D, Hua-hua C, Wei-kang G (2005) Neural network and genetic algorithm based global path planning in a static environment. J Zhejiang Univ Sci A 6(6):549–554

    Google Scholar 

  28. Rimon E, Koditschek DE (1992) Exact robot navigation using artificial potential functions. IEEE Trans Robot Autom 8(5):501–518

    Article  Google Scholar 

  29. Ge SS, Cui YJ (2000) New potential functions for mobile robot path planning. IEEE Trans Robot Autom 16(5):615–620

    Article  Google Scholar 

  30. Guan-Zheng T, Huan H, Sloman A (2007) Ant colony system algorithm for real-time globally optimal path planning of mobile robots. Acta Autom Sin 33(3):279–285

    Article  MATH  Google Scholar 

  31. Sariff NB, Buniyamin N (2009) Comparative study of genetic algorithm and ant colony optimization algorithm performances for robot path planning in global static environments of different complexities. In: CIRA

  32. Brand M et al (2010) Ant colony optimization algorithm for robot path planning. In: 2010 international conference on computer design and applications (ICCDA)

  33. Kala R et al (2009) Mobile robot navigation control in moving obstacle environment using genetic algorithm, artificial neural networks and A* algorithm. In: 2009 WRI world congress on computer science and information engineering

  34. Zeng C, Zhang Q, Wei X (2011) Robotic global path-planning based modified genetic algorithm and A* algorithm. In: 2011 third international conference on measuring technology and mechatronics automation (ICMTMA)

  35. Kang HI, Lee B, Kim K (2008) Path planning algorithm using the particle swarm optimization and the improved Dijkstra algorithm. In: Pacific-Asia workshop on computational intelligence and industrial application, 2008 (PACIIA’08)

  36. Das P, Behera H, Panigrahi B (2015) Intelligent-based multi-robot path planning inspired by improved classical Q-learning and improved particle swarm optimization with perturbed velocity. Int J Eng Sci Technol 19:651–669

  37. Li Y, Li C, Zhang Z (2006) Q-learning based method of adaptive path planning for mobile robot. In: 2006 IEEE international conference on information acquisition

  38. Banerjee D, et al (2012) Path-planning of mobile agent using Q-learning and real-time communication in an unfavourable situation. In: 2012 world congress on information and communication technologies (WICT)

  39. Li S, Xu X, Zuo L (2015) Dynamic path planning of a mobile robot with improved Q-learning algorithm. In: 2015 IEEE international conference on information and automation

  40. Meng Y et al (2011) Cross-ball: a new morphogenetic self-reconfigurable modular robot. In: 2011 IEEE international conference on robotics and automation (ICRA)

  41. Kuffner JJ, LaValle SM (2000) RRT-connect: an efficient approach to single-query path planning. In: Proceedings of IEEE international conference on robotics and automation, 2000 (ICRA’00)

  42. Desaraju VR, How JP (2011) Decentralized path planning for multi-agent teams in complex environments using rapidly-exploring random trees. In: 2011 IEEE international conference on robotics and automation (ICRA)

  43. Bai W, Xue B, Sun Y (2011) Research on path planning for soccer robot based on improved genenic algorithm. In: 2011 international conference on mechatronic science, electric engineering and computer (MEC)

  44. Li Q et al (2006) An improved adaptive algorithm for controlling the probabilities of crossover and mutation based on a fuzzy control strategy. In: Sixth international conference on hybrid intelligent systems, 2006 (HIS’06)

  45. Parlaktuna O, Sipahioğlu A, Yazici A (2007) A VRP-based route planning for a mobile robot group. Turk J Electr Eng Comput Sci 15(2):187–197

    Google Scholar 

  46. Sun X, Druzdzel, MJ, Yuan C (2007) Dynamic weighting A* search-based MAP algorithm for bayesian networks. In: IJCAI

  47. Rosyidi L, et al (2014) Timebase dynamic weight for Dijkstra Algorithm implementation in route planning software. In: 2014 international conference on intelligent green building and smart grid (IGBSG)

  48. Bozdoğan AÖ, Yilmaz AE, Efe M (2010) Performance analysis of swarm optimization approaches for the generalized assignment problem in multi-target tracking applications. Turk J Electr Eng Comput Sci 18(6):1059–1078

    Google Scholar 

  49. Chia S-H et al (2010) Ant colony system based mobile robot path planning. In: 2010 fourth international conference on genetic and evolutionary computing (ICGEC)

  50. Dorigo M, Maniezzo V, Colorni A (1996) Ant system: optimization by a colony of cooperating agents. IEEE Trans Syst Man Cybern Part B (Cybern) 26(1):29–41

    Article  Google Scholar 

  51. Kaelbling LP, Littman ML, Moore AW (1996) Reinforcement learning: a survey. J Artif Intell Res 4:237–285

  52. Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. Bradford Book, Cambridge

    Google Scholar 

  53. Watkins CJ, Dayan P (1992) Q-learning. Mach Learn 8(3–4):279–292

    MATH  Google Scholar 

  54. Huang B-Q, Cao G-Y, Guo M (2005) Reinforcement learning neural network to the problem of autonomous mobile robot obstacle avoidance. In: Proceedings of 2005 international conference on machine learning and cybernetics, 2005

  55. Gao Q, et al (2006) An improved q-learning algorithm based on exploration region expansion strategy. In: Intelligent Control and Automation, 2006. WCICA 2006. The Sixth World Congress on. IEEE

  56. Tokic M (2010) Adaptive \(\varepsilon \)-greedy exploration in reinforcement learning based on value differences. In: Annual conference on artificial intelligence. Springer

  57. Sutton RS, Barto AG (1998) Introduction to reinforcement learning., vol 135. MIT Press, Cambridge

    Google Scholar 

Download references

Acknowledgements

The authors would like to thank Sina Maleki, Salman Faraji and Hossein Davari for their generous help in software design of the robot.

Author information

Authors and Affiliations

  1. Sharif University of Technology, Tehran, Islamic Republic of Iran

    Sajad Haghzad Klidbary, Saeed Bagheri Shouraki & Soroush Sheikhpour Kourabbaslou

Authors
  1. Sajad Haghzad Klidbary
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Saeed Bagheri Shouraki
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Soroush Sheikhpour Kourabbaslou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sajad Haghzad Klidbary.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Haghzad Klidbary, S., Bagheri Shouraki, S. & Sheikhpour Kourabbaslou, S. Path planning of modular robots on various terrains using Q-learning versus optimization algorithms. Intel Serv Robotics 10, 121–136 (2017). https://doi.org/10.1007/s11370-017-0217-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11370-017-0217-x

Keywords

Search

Navigation