On Speeding up the Levenberg-Marquardt Learning Algorithm | SpringerLink
Skip to main content
Springer Nature Link
Log in

On Speeding up the Levenberg-Marquardt Learning Algorithm

  • Conference paper
  • First Online:
Artificial Intelligence and Soft Computing (ICAISC 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14125))

Included in the following conference series:

  • 571 Accesses

Abstract

A new approach to the practical realizations of calculations to the Levenberg-Marquardt learning algorithm is presented. The proposed solutions aim to effectively reduce the high computational load of the LM algorithm. The detailed application of proposed methods in the process of learning neural networks is explicitly discussed. Experimental results have been obtained for all proposed methods and they confirm a very good performance of them.

This work has been supported by the Polish National Science Center under Grant 2017/27/B/ST6/02852.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 10295
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 12869
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bartczuk, Ł., Przybył, A., Cpałka, K.: A new approach to nonlinear modelling of dynamic systems based on fuzzy rules. Int. J. Appl. Math. Comput. Sci. (AMCS) 26(3), 603–621 (2016)

    Google Scholar 

  2. Bilski, J., Kowalczyk, B., Smola̧g, J., Grzanek, K., Izonin, I.: Fast computational approach to the Levenberg-Marquardt algorithm for training feedforward neural networks. J. Artif. Intell. Soft Comput. Res. 12(2), 45–61 (2023)

    Google Scholar 

  3. Bilski, J., Litwiński, S., Smola̧g, J.: Parallel realisation of QR algorithm for neural networks learning. In: Rutkowski, L., Siekmann, J.H., Tadeusiewicz, R., Zadeh, L.A. (eds.) ICAISC 2004. LNCS (LNAI), vol. 3070, pp. 158–165. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24844-6_19

  4. Bilski, J.: The UD RLS algorithm for training the feedforward neural networks. Int. J. Appl. Math. Comput. Sci. 15(1), 101–109 (2005)

    MATH  Google Scholar 

  5. Bilski, J., Smola̧g, J.: Parallel realisation of the recurrent RTRN neural network learning. In: Rutkowski, L., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2008. LNCS (LNAI), vol. 5097, pp. 11–16. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-69731-2_2

    Chapter  Google Scholar 

  6. Bilski, J., Smola̧g, J.: Parallel realisation of the recurrent Elman neural network learning. In: Rutkowski, L., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2010. LNCS (LNAI), vol. 6114, pp. 19–25. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-13232-2_3

    Chapter  Google Scholar 

  7. Bilski, J., Smola̧g, J.: Parallel realisation of the recurrent multi layer perceptron learning. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2012. LNCS (LNAI), vol. 7267, pp. 12–20. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-29347-4_2

  8. Bilski, J., Smola̧g, J.: Parallel approach to learning of the recurrent Jordan neural network. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2013. LNCS (LNAI), vol. 7894, pp. 32–40. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38658-9_3

  9. Bilski, J.: Parallel Structures for Feedforward and Dynamical Neural Networks. AOW EXIT (2013). (in Polish)

    Google Scholar 

  10. Bilski, J., Smola̧g, J., Galushkin, A.I.: The parallel approach to the conjugate gradient learning algorithm for the feedforward neural networks. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2014. LNCS (LNAI), vol. 8467, pp. 12–21. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07173-2_2

  11. Bilski, J., Smola̧g, J.: Parallel architectures for learning the RTRN and Elman dynamic neural networks. IEEE Trans. Parallel Distrib. Syst. 26(9), 2561–2570 (2014). https://doi.org/10.1109/TPDS.2014.2357019

  12. Bilski, J., Kowalczyk, B., Marchlewska, A., Żurada, J.M.: Local Levenberg-Marquardt algorithm for learning feedforward neural networks. J. Artif. Intell. Soft Comput. Res. 10(4), 299–316 (2020). https://doi.org/10.2478/jaiscr-2020-0020

    Article  Google Scholar 

  13. Bilski, J., Kowalczyk, B., Marjański, A., Gandor, M., Żurada, J.M.: A novel fast feedforward neural networks training algorithm. J. Artif. Intell. Soft Comput. Res. 11(4), 287–306 (2021). https://doi.org/10.2478/jaiscr-2021-0017

    Article  Google Scholar 

  14. Bilski, J., Kowalczyk, B., Kisiel-Dorohinicki, M., Siwocha, A., Żurada, J.M.: Towards a very fast feedforward multilayer neural networks training algorithm. J. Artif. Intell. Soft Comput. Res. 12(3), 181–195 (2022). https://doi.org/10.2478/jaiscr-2022-0012

    Article  Google Scholar 

  15. Bilski, J., Rutkowski, L., Smola̧g, J., Tao, D.: A novel method for speed training acceleration of recurrent neural networks. Inf. Sci. 553, 266–279 (2021). https://doi.org/10.1016/j.ins.2020.10.025

  16. Chu, J.L., Krzyżak, A.: The recognition of partially occluded objects with support vector machines, convolutional neural networks, and deep belief networks. J. Artif. Intell. Soft Comput. Res. 4(1), 5–19 (2014)

    Article  Google Scholar 

  17. Cpałka, K., Rutkowski, L.: Flexible Takagi-Sugeno fuzzy systems. In: Proceedings of the International Joint Conference on Neural Networks, Montreal, pp. 1764–1769 (2005)

    Google Scholar 

  18. Cpałka, K., Łapa, K., Przybył, A., Zalasiński, M.: A new method for designing neuro-fuzzy systems for nonlinear modelling with interpretability aspects. Neurocomputing 135, 203–217 (2014)

    Article  Google Scholar 

  19. Cpałka, K., Rebrova, O., Nowicki, R., et al.: On design of flexible neuro-fuzzy systems for nonlinear modelling. Int. J. Gen. Syst. 42(6), Special Issue: SI, 706–720 (2013)

    Google Scholar 

  20. Fahlman, S.: Faster learning variations on backpropagation: an empirical study. In: Proceedings of Connectionist Models Summer School, Los Atos (1988)

    Google Scholar 

  21. Dziwiński, P., Przybył, A., Trippner, P., Paszkowski, J., Havashi, Y.: Hardware implementation of a Takagi-Sugeno neuro-fuzzy system optimized by a population algorithm. J. Artif. Intell. Soft Comput. Res. 11(3), 243–266 (2021). https://doi.org/10.2478/jaiscr-2021-0015

    Article  Google Scholar 

  22. Gabryel, M., Przybyszewski, K.: Methods of searching for similar device fingerprints using changes in unstable parameters. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds.) ICAISC 2020. LNCS (LNAI), vol. 12416, pp. 325–335. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-61534-5_29

    Chapter  Google Scholar 

  23. Gabryel, M., Scherer, M.M., Sułkowski, Ł, Damaševičius, R.: Decision making support system for managing advertisers by Ad fraud detection. J. Artif. Intell. Soft Comput. Res. 11, 331–339 (2021)

    Article  Google Scholar 

  24. Gabryel, M., Kocić, M.: Application of a neural network to generate the hash code for a device fingerprint. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds.) ICAISC 2021. LNCS (LNAI), vol. 12855, pp. 456–463. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87897-9_40

    Chapter  Google Scholar 

  25. Hagan, M.T., Menhaj, M.B.: Training feedforward networks with the Marquardt algorithm. IEEE Trans. Neural Networks 5(6), 989–993 (1994)

    Article  Google Scholar 

  26. Kopczyński, M., Grzes, T.: Hardware rough set processor parallel architecture in FPGA for finding core in big datasets. J. Artif. Intell. Soft Comput. Res. 11(2), 99–110 (2021)

    Article  Google Scholar 

  27. Korytkowski, M., Rutkowski, L., Scherer, R.: From ensemble of fuzzy classifiers to single fuzzy rule base classifier. LNAI 5097, 265–272 (2008)

    Google Scholar 

  28. Korytkowski, M., Scherer, R.: Negative correlation learning of neuro-fuzzy system. LNAI 6113, 114–119 (2010)

    Google Scholar 

  29. Kordos, M., Blachnik, M., Scherer, R.: Fuzzy clustering decomposition of genetic algorithm-based instance selection for regression problems. Inf. Sci. 587, 23–40 (2021)

    Article  Google Scholar 

  30. Łapa, K., Przybył, A., Cpałka, K.: A new approach to designing interpretable models of dynamic systems. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2013. LNCS (LNAI), vol. 7895, pp. 523–534. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38610-7_48

    Chapter  Google Scholar 

  31. Łapa, K., Zalasiński, M., Cpałka, K.: A new method for designing and complexity reduction of neuro-fuzzy systems for nonlinear modelling. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2013. LNCS (LNAI), vol. 7894, pp. 329–344. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38658-9_30

    Chapter  Google Scholar 

  32. Marqardt, D.: An algorithm for last-squares estimation of nonlinear parameters. J. Soc. Ind. Appl. Math. 11, 431–441 (1963)

    Article  Google Scholar 

  33. Niksa-Rynkiewicz, T., Szewczuk-Krypa, N., Witkowska, A., Cpałka, K., Zalasiński, M., Cader, A.: Monitoring regenerative heat exchanger in steam power plant by making use of the recurrent neural network. J. Artif. Intell. Soft Comput. Res. 11(2), 143–155 (2021). https://doi.org/10.2478/jaiscr-2021-0009

    Article  Google Scholar 

  34. Patan, K., Patan, M.: Optimal training strategies for locally recurrent neural networks. J. Artif. Intell. Soft Comput. Res. 1(2), 103–114 (2011)

    MATH  Google Scholar 

  35. Riedmiller, M., Braun, H.: A direct method for faster backpropagation learning: the RPROP algorithm. In: IEEE International Conference on Neural Networks, San Francisco (1993)

    Google Scholar 

  36. Romaszewski, M., Gawron, P., Opozda, S.: Dimensionality reduction of dynamic mesh animations using HO-SVD. J. Artif. Intell. Soft Comput. Res. 3(3), 277–289 (2013)

    Article  Google Scholar 

  37. Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning internal representations by error propagation. In: Rumelhart, D.E., McCelland, J. (eds.) Parallel Distributed Processing, vol. 1, Chapter 8. The MIT Press, Cambridge, Massachusetts, (1986)

    Google Scholar 

  38. Rutkowski, L.: Multiple Fourier series procedures for extraction of nonlinear regressions from noisy data. IEEE Trans. Signal Process. 41(10), 3062–3065 (1993)

    Article  MATH  Google Scholar 

  39. Rutkowski, L.: Identification of MISO nonlinear regressions in the presence of a wide class of disturbances. IEEE Trans. Inf. Theory 37(1), 214–216 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  40. Rutkowski, L., Jaworski, M., Pietruczuk, L., Duda, P.: Decision trees for mining data streams based on the gaussian approximation. IEEE Trans. Knowl. Data Eng. 26(1), 108–119 (2014)

    Article  MATH  Google Scholar 

  41. Rutkowski, L., Przybył, A., Cpałka, K., Er, M.J.: Online speed profile generation for industrial machine tool based on neuro-fuzzy approach. In: Rutkowski, L., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2010. LNCS (LNAI), vol. 6114, pp. 645–650. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-13232-2_79

    Chapter  Google Scholar 

  42. Rutkowski, L., Rafajlowicz, E.: On optimal global rate of convergence of some nonparametric identification procedures. IEEE Trans. Autom. Control 34(10), 1089–1091 (1989)

    Article  MathSciNet  MATH  Google Scholar 

  43. Rutkowski, T., Łapa, K., Jaworski, M., Nielek, R., Rutkowska, D.: On explainable flexible fuzzy recommender and its performance evaluation using the Akaike information criterion. In: Gedeon, T., Wong, K.W., Lee, M. (eds.) ICONIP 2019. CCIS, vol. 1142, pp. 717–724. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-36808-1_78

    Chapter  Google Scholar 

  44. Smola̧g, J., Bilski, J.: A systolic array for fast learning of neural networks. In: Proceedings of V Conference on Neural Networks and Soft Computing, Zakopane, pp. 754–758 (2000)

    Google Scholar 

  45. Smola̧g, J., Rutkowski, L., Bilski, J.: Systolic array for neural networks. In: Proceedings of IV Conference on Neural Networks and Their Applications, Zakopane, pp. 487–497 (1999)

    Google Scholar 

  46. Starczewski, A.: A clustering method based on the modified RS validity index. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2013. LNCS (LNAI), vol. 7895, pp. 242–250. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38610-7_23

    Chapter  Google Scholar 

  47. Starczewski, J.T.: Advanced Concepts in Fuzzy Logic and Systems with Membership Uncertainty, vol. 284. Studies in Fuzziness and Soft Computing. Springer, Cham (2013). https://doi.org/10.1007/978-3-642-29520-1

  48. Starczewski, J.T., Goetzen, P., Napoli, Ch.: Triangular fuzzy-rough set based fuzzification of fuzzy rule-based systems. J. Artif. Intell. Soft Comput. Res. 10(4), 271–285 (2020)

    Article  Google Scholar 

  49. Tadeusiewicz, R.: Neural Networks. AOW RM (1993). (in Polish)

    Google Scholar 

  50. Werbos, J.: Backpropagation through time: what it does and how to do it. In: Proceedings of the IEEE, vol. 78, no. 10 (1990)

    Google Scholar 

  51. Wilamowski, B.M., Yo, H.: Neural network learning without backpropagation. IEEE Trans. Neural Networks 21(11), 1793–1803 (2010)

    Article  Google Scholar 

  52. Zalasiński, M., Cpałka, K.: New approach for the on-line signature verification based on method of horizontal partitioning. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2013. LNCS (LNAI), vol. 7895, pp. 342–350. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38610-7_32

    Chapter  Google Scholar 

  53. Zalasiński, M., Łapa, K., Cpałka, K.: Prediction of values of the dynamic signature features. Expert Syst. Appl. 104, 86–96 (2018)

    Article  Google Scholar 

  54. El Zini, J., Rizk, Y., Awad, M.: An optimized parallel implementation of non-iteratively trained recurrent neural networks. J. Artif. Intell. Soft Comput. Res. 11(1), 33–50 (2021). https://doi.org/10.2478/jaiscr-2021-0003

  55. Sun, Z., Zhao, Z., Scherer, R., Wei, W., Woźniak, M.: An overview of capsule neural networks. J. Internet Technol. 23(1), 33–44 (2022)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computational Intelligence, Czȩstochowa University of Technology, Czȩstochowa, Poland

    Jarosław Bilski, Barosz Kowalczyk & Jacek Smola̧g

Authors
  1. Jarosław Bilski
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Barosz Kowalczyk
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jacek Smola̧g
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jarosław Bilski .

Editor information

Editors and Affiliations

  1. Systems Research Institute of the Polish Academy of Sciences, Warsaw, Poland

    Leszek Rutkowski

  2. Częstochowa University of Technology, Częstochowa, Poland

    Rafał Scherer

  3. Częstochowa University of Technology, Częstochowa, Poland

    Marcin Korytkowski

  4. University of Alberta, Edmonton, AB, Canada

    Witold Pedrycz

  5. AGH University of Krakow, Kraków, Poland

    Ryszard Tadeusiewicz

  6. University of Louisville, Louisville, KY, USA

    Jacek M. Zurada

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bilski, J., Kowalczyk, B., Smola̧g, J. (2023). On Speeding up the Levenberg-Marquardt Learning Algorithm. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds) Artificial Intelligence and Soft Computing. ICAISC 2023. Lecture Notes in Computer Science(), vol 14125. Springer, Cham. https://doi.org/10.1007/978-3-031-42505-9_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-42505-9_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-42504-2

  • Online ISBN: 978-3-031-42505-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics