Abstract
In terms of \(L_{1/2}\) regularization, a novel feature selection method for a neural framework model has been developed in this paper. Due to the non-convex, non-smooth and non-Lipschitz characteristics of \(L_{1/2}\) regularizer, it is difficult to directly employ the gradient descent method in training multilayer perceptron neural networks. A smoothing technique has been considered to approximate the original \(L_{1/2}\) regularizer. The proposed method is a two-stage updating approach. First, a multilayer network model with smoothing \(L_{1/2}\) regularizer is trained to eliminate the unimportant features. Second, the compact model without regularization has been simulated until there is no improvements for the performance. The experiments demonstrate that the presented algorithm significantly reduces the redundant features while keeps a considerable model accuracy.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Jain, A.K., Duin, R.P.W., Mao, J.: Statistical pattern recognition: a review. IEEE Trans. Pattern Anal. Mach. Intell. 22, 4–37 (2000)
Sun, Y., Todorovic, S., Goodison, S.: Local-learning-based feature selection for high-dimensional data analysis. IEEE Trans. Pattern Anal. Mach. Intell. 32, 1610–1626 (2010)
Liu, H., Yu, L.: Toward Integrating Feature Selection Algorithms for Classification and Clustering. IEEE Educational Activities Department (2005)
Bu, H.L., Xia, J., Han, J.B.: A summary of feature selection algorithms and prospect. J. Chaohu Coll. (2008)
Chakraborty, R., Pal, N.R.: Feature selection using a neural framework with controlled redundancy. IEEE Trans. Neural Netw. Learn. Syst. 26, 35–50 (2014)
He, X., Cai, D., Niyogi, P.: Laplacian score for feature selection. In: International Conference on Neural Information Processing Systems, pp. 507–514. MIT Press (2005)
Gu, Q., Li, Z., Han, J.: Generalized Fisher Score for Feature Selection, pp. 266–273 (2012)
Yao, Y.: Statistical Applications of Linear Programming for Feature Selection via Regularization Methods (2008)
Burden, F., Winkler, D.: Bayesian regularization of neural networks. Methods Mol. Biol. 458, 25 (2008)
Ticknor, J.L.: A Bayesian regularized artificial neural network for stock market forecasting. Expert Syst. Appl. 40, 5501–5506 (2013)
Han, M., Li, D.: An norm 1 regularization term ELM algorithm based on surrogate function and Bayesian framework. Acta Autom. Sinica 37, 1344–1350 (2011)
Tibshirani, R.J.: Regression shrinkage and selection via the LASSO. J. Roy. Stat. Soc. Ser. B 58, 267–288 (1996)
Xu, Z.B., Guo, H.L., Wang, Y., Zhang, H.: Representative of L1/2 regularization among Lq (0 \(\le \) q \(\le \)1) regularizations: an experimental study based on phase diagram. Acta Autom. Sinica 38, 1225–1228 (2012)
Liu, C., Liang, Y., Luan, X.Z., Leung, K.S., Chan, T.M., Xu, Z.B., Zhang, H.: A improve direct path seeking algorithm for L1/2 regularization, with application to biological feature selection. In: International Conference on Biomedical Engineering and Biotechnology, pp. 8–11. IEEE (2012)
Wu, W., Yang, J.: L1/2 regularization methods for weights sparsication of neural networks (in Chinese). Sci. Sinica 45, 1487–1504 (2015)
Wu, W., Shao, H., Li, Z.: Convergence of batch BP algorithm with penalty for FNN training. In: King, I., Wang, J., Chan, L.-W., Wang, D.L. (eds.) ICONIP 2006. LNCS, vol. 4232, pp. 562–569. Springer, Heidelberg (2006). doi:10.1007/11893028_63
Haykin, S.: Neural Networks: A Comprehensive Foundation. Prentice-Hall, Upper Saddle River (1994)
Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators. Neural Netw. 2, 359–366 (1989)
Kim, Y., Kim, J.: Gradient LASSO for feature selection. In: International Conference on Machine Learning, p. 60. ACM (2004)
Challita, N., Khalil, M., Beauseroy, P.: New feature selection method based on neural network and machine learning. In: Multidisciplinary Conference on Engineering Technology, pp. 81–85. IEEE (2016)
Setiono, R.: A penalty-function approach for pruning feedforward neural networks. Neural Comput. 9, 185 (1997)
Sun, K., Huang, S.H., Wong, D.S., Jang, S.S.: Design and application of a variable selection method for multilayer perceptron neural network with LASSO. IEEE Trans. Neural Netw. Learn. Syst. 28, 1386–1396 (2017)
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China (Nos. 61305075, 11401185), the China Postdoctoral Science Foundation (No. 2012M520624), the Natural Science Foundation of Shandong Province (Nos. ZR2013FQ004, ZR2013DM015, ZR2015AL014), the Specialized Research Fund for the Doctoral Program of Higher Education of China (No. 20130133120014), the Fundamental Research Funds for the Central Universities (Nos. 14CX05042A, 15CX05053A, 15CX08011A, 15CX02064A) and the University-level Undergraduate Training Program for Innovation and Entrepreneurship (No. 20161349).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Gao, H., Yang, Y., Zhang, B., Li, L., Zhang, H., Wu, S. (2017). Feature Selection Using Smooth Gradient \(L_{1/2}\) Regularization. In: Liu, D., Xie, S., Li, Y., Zhao, D., El-Alfy, ES. (eds) Neural Information Processing. ICONIP 2017. Lecture Notes in Computer Science(), vol 10637. Springer, Cham. https://doi.org/10.1007/978-3-319-70093-9_17
Download citation
DOI: https://doi.org/10.1007/978-3-319-70093-9_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-70092-2
Online ISBN: 978-3-319-70093-9
eBook Packages: Computer ScienceComputer Science (R0)