Search for the Global Extremum Using the Correlation Indicator for Neural Networks Supervised Learning | Programming and Computer Software Skip to main content
Springer Nature Link
Log in

Search for the Global Extremum Using the Correlation Indicator for Neural Networks Supervised Learning

  • Published:
Programming and Computer Software Aims and scope Submit manuscript

Abstract

The article discusses the search for a global extremum in the training of artificial neural networks using a correlation indicator. A method based on a mathematical model of an artificial neural network presented as an information transmission system is proposed. Drawing attention to the fact that in information transmission systems widely used methods that allow effective analysis and recovery of useful signal against the background of various interferences: Gaussian, concentrated, pulsed, etc., it is possible to make an assumption about the effectiveness of the mathematical model of artificial neural network, presented as a system of information transmission. The article analyzes the convergence of training and experimentally obtained sequences based on a correlation indicator for fully-connected neural network. The possibility of estimating the convergence of the training and experimentally obtained sequences based on the joint correlation function as a measure of their energy similarity (difference) is confirmed. To evaluate the proposed method, a comparative analysis is made with the currently used indicators. The potential sources of errors in the least-squares method and the possibilities of the proposed indicator to overcome them are investigated. Simulation of the learning process of an artificial neural network has shown that the use of the joint correlation function together with the Adadelta optimizer allows us to get again in learning speed 2-3 times compared to CrossEntropyLoss.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

REFERENCES

  1. Kolmogorov, A.N., On the representation of continuous functions of many variables by superposition of continuous functions of one variable and addition, Am. Math. Soc. Transl., Ser. 2, 1963, vol. 28, pp. 55–59.

    MATH  Google Scholar 

  2. Arnol’d, V.I., On the representation of functions of several variables by superpositions of functions of fewer variables, Mat. Prosveshchenie., 1958, no. 3, pp. 41–61.

  3. Hecht-Nielsen, R., Neurocomputing, Addison-Wesley, 1989.

    Google Scholar 

  4. Dzyadyk, V.K., Introduction to the Theory of the Uniform Approximation of Functions by Polynomials, Moscow: Nauka, 1977.

    MATH  Google Scholar 

  5. Hebb, D.O., The Organization of Behavior, New York: Wiley, 1949.

    Google Scholar 

  6. Hinton, G.E., Training products of experts by minimizing contrastive divergence, Neural Comput., 2002, vol. 14, no. 8, pp. 1771–1800.

    Article  Google Scholar 

  7. Hinton, G.E., Learning multiple layers of representation, Trends Cognit. Sci., 2007, vol. 11, pp. 428–434.

    Article  Google Scholar 

  8. Nuzhny, A.S., Bayes regularization in the selection of weight coefficients in the predictor ensembles, Proc. ISP RAS, 2019, vol. 31, no. 4, pp. 113–120. https://doi.org/10.15514/ISPRAS-2019-31(4)-7

  9. García-Hernández, L.E., et al., Multi-objective configuration of a secured distributed cloud data storage, in Proc. 6th Latin American Conf. on High Performance Computing CARLA 2019, Cham: Springer, 2019, vol. 1087. https://doi.org/10.1007/978-3-030-41005-6_6

  10. Nikolenko, S., Kadurin, A., and Arhangel’skaya, E., Deep Learning, St. Petersburg: Piter, 2018.

    Google Scholar 

  11. Dorogov, A.Y., Implementation of spectral transformations in the class of fast neural networks, Program. Comput. Software, 2003, vol. 29, no. 4, pp. 187–198.

    Article  MathSciNet  Google Scholar 

  12. Adjemov, S.S., et al., The use of artificial neural networks for classification of signal sources in cognitive radio systems, Program. Comput. Software, 2016, vol. 42, no. 3, pp. 121–128.

    Article  MathSciNet  Google Scholar 

  13. Vershkov, N.A., Kuchukov, V.A., Kuchukova, N.N., and Babenko, M., The wave model of artificial neural network, Proc. IEEE Conf. of Russian Young Researchers in Electrical and Electronic Engineering, EIConRus 2020, Moscow, St. Petersburg, 2020, pp. 542–547. https://doi.org/10.1109/EIConRus49466.2020.9039172

  14. Shannon, C., Works on Information Theory and Cybernetics, Moscow: Izd. inostrannoi literatury, 1963.

  15. Sikarev, A.A. and Lebedev, O.N., Microelectronic Devices for the Generation and Processing of Complex Signals, Moscow: Radio i svyaz’, 1983.

  16. Widrow, B., Adaptive sampled-data systems, a statistical theory of adaptation, in IRE WESCON Convention Record, New York: Institute of Radio Engineers, 1959, part 4.

  17. Ifeachor, E.C. and Jervis, B.W., Digital Signal Processing: a Practical Approach, Pearson Education, 2002.

    Google Scholar 

  18. Solodov, A.V., Information Theory and Its Application to Tasks of Automatic Control and Monitoring, Moscow: Nauka, 1967.

    Google Scholar 

  19. Schmidhuber, J., Deep learning in neural networks: an overview, Neural Networks, 2015, vol. 1, pp. 85–117. https://doi.org/10.1016/j.neunet.2014.09.003

    Article  Google Scholar 

  20. Erofeeva, V.A., An overview of data mining concepts based on neural networks, Stohasticheskaya Optim. Inf., 2015, vol. 11, no. 3, pp. 3–17.

    Google Scholar 

  21. Cypkin, Y.A., Information Theory of Identification, Moscow: Nauka, Fizmatlit, 1995.

    Google Scholar 

  22. Vershkov, N.N., Kuchukov, V.A., and Kuchukova, N.N., The theoretical approach to the search for a global extremum in the training of neural networks, Proc. ISP RAS, 2019, vol. 31, no. 2, pp. 41–52.

  23. Haykin. S., Neural Networks: a Comprehensive Foundation, Prentice Hall, 1999.

  24. Linnik, Yu.V., The Method of Least Squares and Basics of Mathematical-Statistical Theory of Processing Observations, Moscow: Gos. izd. fizikomatematich. lit., 1958.

  25. Rao, D. and McMahan, B., Natural Language Processing with PyTorch: Build Intelligent Language Applications Using Deep Learning, O’Reilly Media Inc., 2019.

    Google Scholar 

  26. LeCun, Y., The MNIST database of handwritten digits, 1998. http://yann.lecun.com/exdb/mnist/. Accessed 03.11.19.

  27. PyTorch. https://pytorch.org/. Accessed 03.11.19.

Download references

ACKNOWLEDGMENTS

The reported study was funded by RFBR, project number 20-37-70023, and Russian Federation President Grant MK-341.2019.9 and SP-2236.2018.5.

Author information

Authors and Affiliations

  1. North-Caucasus Federal University, 355029, Stavropol, Russia

    N. Vershkov, M. Babenko, V. Kuchukov & N. Kuchukova

Authors
  1. N. Vershkov
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. M. Babenko
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. V. Kuchukov
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. N. Kuchukova
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding authors

Correspondence to N. Vershkov, M. Babenko, V. Kuchukov or N. Kuchukova.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vershkov, N., Babenko, M., Kuchukov, V. et al. Search for the Global Extremum Using the Correlation Indicator for Neural Networks Supervised Learning. Program Comput Soft 46, 609–618 (2020). https://doi.org/10.1134/S0361768820080265

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1134/S0361768820080265