On a Streaming Approach for Training Denoising Auto-encoders | SpringerLink
Skip to main content
Springer Nature Link
Log in

On a Streaming Approach for Training Denoising Auto-encoders

  • Conference paper
  • First Online:
Artificial Intelligence and Soft Computing (ICAISC 2020)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12416))

Included in the following conference series:

Abstract

Learning deep neural networks requires huge hardware resources and takes a long time. This is due to the need to process huge data sets multiple times. One type of neural networks that are particularly useful in practice are denoising autoencoders. It is, therefore, necessary to create new algorithms that reduce the training time for this type of networks. In this work, we propose a method that, in contrast to the classical approach, where each data element is repeatedly processed by the network, is focused on processing only the most difficult to analyze elements. In the learning process, subsequent data may lose their significance and others may become important. Therefore, an additional algorithm has been used to detect such changes. The method draws inspiration from boosting algorithms and drift detectors.

This work was supported by the Polish National Science Centre under grant no. 2017/27/B/ST6/02852.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 5719
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 7149
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Akdeniz, E., Egrioglu, E., Bas, E., Yolcu, U.: An ARMA type pi-sigma artificial neural network for nonlinear time series forecasting. J. Artif. Intell. Soft Comput. Res. 8(2), 121–132 (2018)

    Google Scholar 

  2. Bengio, Y.: Learning deep architectures for AI. Found. Trends Mach. Learn. 2(1), 1–127 (2009)

    Article  MATH  Google Scholar 

  3. Bifet, A., Gavaldà, R.: Adaptive learning from evolving data streams. In: Adams, N.M., Robardet, C., Siebes, A., Boulicaut, J.-F. (eds.) IDA 2009. LNCS, vol. 5772, pp. 249–260. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-03915-7_22

    Chapter  Google Scholar 

  4. Bodyanskiy, Y., Vynokurova, O., Pliss, I., Setlak, G, Mulesa, P.: Fast learning algorithm for deep evolving GMDH-SVM neural network in data stream mining tasks. In: 2016 IEEE First International Conference on Data Stream Mining Processing (DSMP), pp. 257–262, August 2016

    Google Scholar 

  5. deBarros, R.S.M., Hidalgo, J.I.G., de Lima Cabral, D.R.: Wilcoxon rank sum test drift detector. Neurocomputing 275, 1954–1963 (2018)

    Google Scholar 

  6. Demsar, J., Bosnic, Z.: Detecting concept drift in data streams using model explanation. Expert Syst. Appl. 92, 546–559 (2018)

    Google Scholar 

  7. Du, B., Xiong, W., Wu, J., Zhang, L., Zhang, L., Tao, D.: Stacked convolutional denoising auto-encoders for feature representation. IEEE Trans. Cybern. 47(4), 1017–1027 (2016)

    Google Scholar 

  8. Duda, P., Jaworski, M., Cader, A., Wang, L.: On training deep neural networks using a streaming approach. J. Artif. Intell. Soft Comput. Res. 10(1), 15–26 (2020)

    Google Scholar 

  9. Duda, P., Jaworski, M., Rutkowski, L.: On ensemble components selection in data streams scenario with reoccurring concept-drift. In: 2017 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1–7. IEEE (2017)

    Google Scholar 

  10. Duda, P., Jaworski, M., Rutkowski, L.: Convergent time-varying regression models for data streams: tracking concept drift by the recursive Parzen-based generalized regression neural networks. Int. J. Neural Syst. 28(02), 1750048 (2018)

    Google Scholar 

  11. Duda, P., Jaworski, M., Rutkowski, L.: Knowledge discovery in data streams with the orthogonal series-based generalized regression neural networks. Inf. Sci. 460, 497–518 (2018)

    MathSciNet  MATH  Google Scholar 

  12. Duda, P., Jaworski, M., Rutkowski, L.: Online GRNN-based ensembles for regression on evolving data streams. In: Huang, T., Lv, J., Sun, C., Tuzikov, A.V. (eds.) ISNN 2018. LNCS, vol. 10878, pp. 221–228. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-92537-0_26

    Chapter  Google Scholar 

  13. Duda, P., Rutkowski, L., Jaworski, M., Rutkowska, D.: On the Parzen kernel-based probability density function learning procedures over time-varying streaming data with applications to pattern classification. IEEE Trans. Cybern. 50(4), 1683–1696 (2020)

    Google Scholar 

  14. Gama, J., Medas, P., Castillo, G., Rodrigues, P.: Learning with drift detection. In: Bazzan, A.L.C., Labidi, S. (eds.) SBIA 2004. LNCS (LNAI), vol. 3171, pp. 286–295. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-28645-5_29

    Chapter  Google Scholar 

  15. Gomes, H.M., et al.: Adaptive random forests for evolving data stream classification. Mach. Learn. 1469–1495 (2017). https://doi.org/10.1007/s10994-017-5642-8

  16. Gondara, L.: Medical image denoising using convolutional denoising autoencoders. In: 2016 IEEE 16th International Conference on Data Mining Workshops (ICDMW), pp. 241–246. IEEE (2016)

    Google Scholar 

  17. Grais, E.M., Plumbley, M.D.: Single channel audio source separation using convolutional denoising autoencoders. In: 2017 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pp. 1265–1269. IEEE (2017)

    Google Scholar 

  18. Hou, Y., Holder, L.B.: On graph mining with deep learning: introducing model r for link weight prediction. J. Artif. Intell. Soft Comput. Res. 9(1), 21–40 (2019)

    Google Scholar 

  19. Jaworski, M.: Regression function and noise variance tracking methods for data streams with concept drift. Int. J. Appl. Math. Comput. Sci. 28(3), 559–567 (2018)

    MathSciNet  MATH  Google Scholar 

  20. Jaworski, M., Duda, P., Rutkowski, L.: New splitting criteria for decision trees in stationary data streams. IEEE Trans. Neural Netw. Learn. Syst. 29(6), 2516–2529 (2017)

    MathSciNet  Google Scholar 

  21. Jaworski, M., Duda, P., Rutkowski, L.: On applying the restricted Boltzmann machine to active concept drift detection. In: 2017 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1–8. IEEE (2017)

    Google Scholar 

  22. Jaworski, M., Duda, P., Rutkowski, L.: Concept drift detection in streams of labelled data using the restricted Boltzmann machine. In: 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1–7. IEEE (2018)

    Google Scholar 

  23. Jaworski, M., Rutkowski, L., Duda, P., Cader, A.: Resource-aware data stream mining using the restricted Boltzmann machine. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds.) ICAISC 2019. LNCS (LNAI), vol. 11509, pp. 384–396. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20915-5_35

    Chapter  Google Scholar 

  24. Kamimura, R.: Supposed maximum mutual information for improving generalization and interpretation of multi-layered neural networks. J. Artif. Intell. Soft Comput. Res. 9(2), 123–147 (2019)

    Google Scholar 

  25. Koren, O., Hallin, C.A., Perel, N., Bendet, D.: Decision-making enhancement in a big data environment: application of the k-means algorithm to mixed data. J. Artif. Intell. Soft Comput. Res. 9(4), 293–302 (2019)

    Google Scholar 

  26. Kumarratneshk, R., Weilleweill, E., Aghdasi, F., Sriram, P.: A strong and efficient baseline for vehicle re-identification using deep triplet embedding. J. Artif. Intell. Soft Comput. Res. 10(1), 27–45 (2020)

    Google Scholar 

  27. Ludwig, S.A.: Applying a neural network ensemble to intrusion detection. J. Artif. Intell. Soft Comput. Res. 9(3), 177–188 (2019)

    Google Scholar 

  28. Nobukawa, S., Nishimura, H., Yamanishi, T.: Pattern classification by spiking neural networks combining self-organized and reward-related spike-timing-dependent plasticity. J. Artif. Intell. Soft Comput. Res. 9(4), 283–291 (2019)

    Article  Google Scholar 

  29. Ororbia, A.G.I., Giles, C.L., Reitter, D.: Online semi-supervised learning with deep hybrid Boltzmann machines and denoising autoencoders. CoRR, abs/1511.06964 (2015)

    Google Scholar 

  30. Page, E.S.: Continuous inspection schemes. Biometrika 41(1/2), 100–115 (1954)

    MathSciNet  MATH  Google Scholar 

  31. Pietruczuk, L., Rutkowski, L., Jaworski, M., Duda, P.: The Parzen kernel approach to learning in non-stationary environment. In: 2014 International Joint Conference on Neural Networks (IJCNN), pp. 3319–3323. IEEE (2014)

    Google Scholar 

  32. Pietruczuk, L., Rutkowski, L., Jaworski, M., Duda, P.: A method for automatic adjustment of ensemble size in stream data mining. In: 2016 International Joint Conference on Neural Networks (IJCNN), pp. 9–15. IEEE (2016)

    Google Scholar 

  33. Pietruczuk, L., Rutkowski, L., Jaworski, M., Duda, P.: How to adjust an ensemble size in stream data mining? Inf. Sci. 381, 46–54 (2017)

    MathSciNet  MATH  Google Scholar 

  34. Rafajłowicz, E., Rafajłowicz, W.: Testing (non-) linearity of distributed-parameter systems from a video sequence. Asian J. Control 12(2), 146–158 (2010)

    MathSciNet  Google Scholar 

  35. Rafajłowicz, E., Rafajłowicz, W.: Iterative learning in repetitive optimal control of linear dynamic processes. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2016. LNCS (LNAI), vol. 9692, pp. 705–717. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-39378-0_60

    Chapter  Google Scholar 

  36. Rafajłowicz, E., Rafajłowicz, W.: Iterative learning in optimal control of linear dynamic processes. Int. J. Control 91(7), 1522–1540 (2018)

    MathSciNet  MATH  Google Scholar 

  37. Rafajłowicz, E., Wnuk, M., Rafajłowicz, W.: Local detection of defects from image sequences. Int. J. Appl. Math. Comput. Sci. 18(4), 581–592 (2008)

    MATH  Google Scholar 

  38. Read, J., Perez-Cruz, F., Bifet, A.: Deep learning in partially-labeled data streams. In: Proceedings of the 30th Annual ACM Symposium on Applied Computing, SAC 2015, New York, NY, USA, pp. 954–959. ACM (2015)

    Google Scholar 

  39. Rutkowski, L., Pietruczuk, L., Duda, P., Jaworski, M.: Decision trees for mining data streams based on the McDiarmid’s bound. IEEE Trans. Knowl. Data Eng. 25(6), 1272–1279 (2013)

    Google Scholar 

  40. Shewalkar, A., Nyavanandi, D., Ludwig, S.A.: Performance evaluation of deep neural networks applied to speech recognition: RNN, LSTM and GRU. J. Artif. Intell. Soft Comput. Res. 9(4), 235–245 (2019)

    Google Scholar 

  41. Wang, L., Zhang, Z., Chen, J.: Short-term electricity price forecasting with stacked denoising autoencoders. IEEE Trans. Power Syst. 32(4), 2673–2681 (2016)

    Google Scholar 

  42. Wu, Y., DuBois, C., Zheng, A.X., Ester, M.: Collaborative denoising auto-encoders for top-n recommender systems. In: Proceedings of the Ninth ACM International Conference on Web Search and Data Mining, pp. 153–162 (2016)

    Google Scholar 

  43. Zalasinski, M., Lapa, K., Cpalka, K., Przybyszewski, K., Yen, G.G.: On-line signature partitioning using a population based algorithm. J. Artif. Intell. Soft Comput. Res. 10(1), 5–13 (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Czestochowa University of Technology, Czestochowa, Poland

    Piotr Duda

  2. Nanyang Technological University, Singapore, Singapore

    Lipo Wang

Authors
  1. Piotr Duda
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lipo Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Piotr Duda .

Editor information

Editors and Affiliations

  1. Częstochowa University of Technology, Częstochowa, Poland

    Leszek Rutkowski

  2. Częstochowa University of Technology, Częstochowa, Poland

    Rafał Scherer

  3. Częstochowa University of Technology, Częstochowa, Poland

    Marcin Korytkowski

  4. Electrical and Computer Engineering, University of Alberta, Edmonton, AB, Canada

    Witold Pedrycz

  5. AGH University of Science and Technology, Kraków, Poland

    Ryszard Tadeusiewicz

  6. Electrical and Computer Engineering, University of Louisville, Louisville, KY, USA

    Jacek M. Zurada

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Duda, P., Wang, L. (2020). On a Streaming Approach for Training Denoising Auto-encoders. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds) Artificial Intelligence and Soft Computing. ICAISC 2020. Lecture Notes in Computer Science(), vol 12416. Springer, Cham. https://doi.org/10.1007/978-3-030-61534-5_28

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-61534-5_28

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-61533-8

  • Online ISBN: 978-3-030-61534-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation