Abstract
The Boltzmann machine (BM) model is able to learn the probability distribution of input patterns. However, in analog realization, there are thermal noise and random offset voltages of amplifiers. Those realization issues affect the behaviour of the neurons’ activation function and they can be modelled as random input drifts. This paper analyzes the activation function and state distribution of BMs under the input random drift model. Since the state of a neuron is also determined by its activation function, the random input drifts may cause a BM to change the behaviour. We show that the effect of random input drifts is equivalent to raising temperature factor. Hence, from the Kullback–Leibler (KL) divergence perspective, we propose a compensation scheme to reduce the effect of random input drifts. In our derive of compensation scheme, we assume that the input drift follows the Gaussian distribution. Surprisedly, from our simulations, the proposed compensation scheme also works very well for other distributions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ackley, D.H., Hinton, G.E., Sejnowski, T.J.: A learning algorithm for Boltzmann machines. Cogn. Sci. 9(1), 147–169 (1985)
Hinton, G.E., Sejnowski, T.J., Ackley, D.H.: Boltzmann machines: constraint satisfaction networks that learn. Carnegie-Mellon University, Pittsburgh (1984)
Prager, R.W., Harrison, T.D., Fallside, F.: Boltzmann machines for speech recognition. Comput. Speech Lang. 1(1), 3–27 (1986)
Ma, H.: Pattern recognition using Boltzmann machine. In: Proceedings IEEE Southeastcon 1995. Visualize the Future, pp. 23–29 (1995)
Tang, Y., Salakhutdinov, R., Hinton, G.: Robust Boltzmann machines for recognition and denoising. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2264–2271 (2012)
Lee, P.: Low noise amplifier selection guide for optimal noise performance. In: Analog Devices Application Note AN-940 (2009)
He, J., Zhan, S., Chen, D., Geiger, R.L.: Analysis of static and dynamic random offset voltages in dynamic comparators. IEEE Trans. Circuits Syst. I Regul. Pap. 56(5), 911–919 (2009)
Redoute, J.-M., Steyaert, M.: Measurement of emi induced input offset voltage of an operational amplifier. Electronics Letters. 43(20), 1088–1090 (2007)
Wang, H., Feng, R., Han, Z.F., Leung, C.S.: ADMM-based algorithm for training fault tolerant RBF networks and selecting centers. IEEE Trans. Neural Netw. Learn. Syst. 29(8), 3870–3878 (2018)
Leung, C.S., Wan, W.Y., Feng, R.: A regularizer approach for RBF networks under the concurrent weight failure situation. IEEE Trans. Neural Netw. Learn. Syst. 28(6), 1360–1372 (2017)
Xiao, Y., Feng, R., Leung, C.S., Sum, J.: Objective function and learning algorithm for the general node fault situation. IEEE Trans. Neural Netw. Learn. Syst. 27(4), 863–874 (2016)
Chen, H., Murray, A.F.: Continuous restricted Boltzmann machine with an implementable training algorithm. IEE Proc. Vis. Image Signal Process. 150(3), 153–158 (2003)
Chen, H., Fleury, P.C.D., Murray, A.F.: Continuous-valued probabilistic behavior in a VLSI generative model. IEEE Trans. Neural Netw. Learn. Syst. 17(3), 755–770 (2006)
Sum, J., Leung, C.S.: Learning algorithm for Boltzmann machines with additive weight and bias noise. IEEE Trans. Neural Netw. Learn. Syst. 30(10), 3200–3204 (2019)
Haley, D.C.: Estimation of the dosage mortality relationship when the dose is subject to error. Tech. rep. Applied Mathematics and Statistics Labs. Standford University (1952)
Acknowledgement
The work presented in this paper is supported by a research grant from the Taiwan MOST No. 108-2221-E-005-036 and a research grant from City University of Hong Kong (9610431).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Lu, W., Leung, CS., Sum, J. (2020). Analysis on the Boltzmann Machine with Random Input Drifts in Activation Function. In: Yang, H., Pasupa, K., Leung, A.CS., Kwok, J.T., Chan, J.H., King, I. (eds) Neural Information Processing. ICONIP 2020. Lecture Notes in Computer Science(), vol 12534. Springer, Cham. https://doi.org/10.1007/978-3-030-63836-8_14
Download citation
DOI: https://doi.org/10.1007/978-3-030-63836-8_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-63835-1
Online ISBN: 978-3-030-63836-8
eBook Packages: Computer ScienceComputer Science (R0)