Relevant feature selection and ensemble classifier design using bi-objective genetic algorithm | Knowledge and Information Systems Skip to main content
Springer Nature Link
Log in

Relevant feature selection and ensemble classifier design using bi-objective genetic algorithm

  • Regular Paper
  • Published:
Knowledge and Information Systems Aims and scope Submit manuscript

Abstract

In the era of digital boom, single classifier cannot perform well in various datasets. Ensemble classifier aims to bridge this performance gap by combining multiple classifiers of diverse characteristics to get better generalization. But classifier selection highly depends on the dataset, and its efficiency degrades tremendously due to the presence of irrelevant features. Feature selection aids the performance of classifier by removing those irrelevant features. Initially, we have proposed a bi-objective genetic algorithm-based feature selection method (FSBOGA), where nonlinear, uniform, hybrid cellular automata are used to generate an initial population. Objective functions are defined using lower bound approximation of rough set theory and Kullback–Leibler divergence method of information theory to select unambiguous and informative features. The replacement strategy for creation of next-generation population is based on the Pareto optimal solution with respect to both the objective functions. Next, a novel bi-objective genetic algorithm-based ensemble classification method (CCBOGA) is devised to ensemble the individual classifiers designed using obtained reduced datasets. It is observed that the constructed ensemble classifier performs better than the individual classifiers. The performances of proposed FSBOGA and CCBOGA are investigated on some popular datasets and compared with the state-of-the-art algorithms to demonstrate their effectiveness.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Abbaszadeh O, Amiri A, Khanteymoori AR (2015) An ensemble method for data stream classification in the presence of concept drift. Front Inf Technol Electron Eng 16(2):1059–1068

    Article  Google Scholar 

  2. Acharyya A, Rakshit S, Sarkar R, Basu S, Nasipuri M (2013) Handwritten word recognition using MLP based classifier: a holistic approach. IJCSI Int J Comput Sci Issues 10(2):422–427

    Google Scholar 

  3. Bache K, Lichman M (2013) UCI machine learning repository, p 901. http://archive.ics.uci.edu/ml. Accessed 2013

  4. Bandyopadhyay S, Bhadra T, Mitra P, Maulik U (2014) Integration of dense sub graph finding with feature clustering for unsupervised feature selection. Pattern Recogn Lett 40:104–112

    Article  Google Scholar 

  5. Bauer E, Kohavi R (1999) An empirical comparison of voting classification algorithms: bagging, boosting, and variants. Mach Learn 35:1–38

    Google Scholar 

  6. Bernstein E, Amit Y (2005) Part-based statistical models for object classification and detection. Proc Comput Vis Pattern Recognit (CVPR) 2:734–740

    Google Scholar 

  7. Cai D, Zhang C, He X (2010) Unsupervised feature selection for multi-cluster data. In: 16th ACM SIGKDD international conference on knowledge discovery and Data mining, pp 333–342

  8. Chaconas G, Lavoie BD, Watson MA (1996) DNA transposition: jumping gene machine. Curr Biol 6(7):817–820

    Article  Google Scholar 

  9. Chandrashekar G, Sahin F (2014) A survey on feature selection methods. Comput Electr Eng 40(1):16–28

    Article  Google Scholar 

  10. Cheng X, Cai H, Zhang Y, Xu B, Su W (2015) Optimal combination of feature selection and classification via local hyperplane based learning strategy. BMC Bioinform 16:219. https://doi.org/10.1186/s12859-015-0629-6

    Article  Google Scholar 

  11. Cheng S, Chen M, Wai R, Wang F (2014) Optimal placement of distributed generation units in distribution systems via an enhanced multi-objective particle swarm optimization algorithm. J Zhejiang Univ Sci 15(4):300–311

    Article  Google Scholar 

  12. Cyganek B (2015) Hybrid ensemble of classifiers for logo and trademark symbols recognition. Soft Comput 19(12):3413–3430

    Article  Google Scholar 

  13. Das AK, Sil J (2011) An efficient classifier design integrating rough set and set oriented database operations. Appl Soft Comput 11:2279–2285

    Article  Google Scholar 

  14. Das AK, Das S, Ghosh A (2017) Ensemble feature selection using bi-objective genetic algorithm. Knowl Based Syst 123:116–127

    Article  Google Scholar 

  15. Deb K, Pratap A, Agarwal S, Meyarivan TA (2002) A fast and elitist multi objective genetic algorithm: NSGA-II. IEEE Trans Evol Comput 6(2):182–197

    Article  Google Scholar 

  16. Dietterich TG (2000) An experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting and randomization. Mach Learn 40(2):139–157

    Article  Google Scholar 

  17. Ding C, Peng H (2005) Minimum redundancy feature selection from microarray gene expression data. J Bioinform Comput Biol 3(2):185–205

    Article  Google Scholar 

  18. Fortino V, Kinaret P, Fyhrquist N, Alenius H, Greco D (2014) A robust and accurate method for feature selection and prioritization from multi-class OMICs data. PLoS ONE 9(9):e107801. https://doi.org/10.1371/journal.pone.0107801

    Article  Google Scholar 

  19. Freund Y, Schapire R (1996) Experiments with new boosting algorithms. In: International conference on machine learning

  20. Gabrys B, Ruta D (2006) Genetic algorithms in classifier fusion. Appl Soft Comput 6(4):337–347

    Article  Google Scholar 

  21. Gu F, Liu HL, Tan KC (2015) A hybrid evolutionary multi-objective optimization algorithm with adaptive multi-fitness assignment. Soft Comput 19(11):3249–3259

    Article  Google Scholar 

  22. Hall AM (1999) Correlation-based feature selection for machine learning. The University of Waikato

  23. Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The WEKA data mining software: an update. SIGKDD Explor 11(1):10–18

    Article  Google Scholar 

  24. Jing SY (2014) A hybrid genetic algorithm for feature subset selection in rough set theory. Soft Comput 18(7):1373–1382

    Article  Google Scholar 

  25. Kent Ridge Bio-medical Data Set Repository. http://datam.i2r.a-star.edu.sg/datasets/krbd. Accessed 2002

  26. Kerber R (1992) ChiMerge: discretization of numeric attributes. In: Tenth national conference on artificial intelligence, pp 123–128

  27. Kim S, Scalzo F, Telesca D, Hu X (2015) Ensemble of sparse classifiers for high-dimensional biological data. Int. J. Data Min Bioinform 12(2):167–183

    Article  Google Scholar 

  28. Knowles JD, Corne DW (2000) M-PAES: a memetic algorithm for multi-objective optimization. In: IEEE congress on evolutionary computation, pp 325–332

  29. Kullback S, Leibler RA (1951) On information and sufficiency. Ann Math Stat 22(1):79–86

    Article  MathSciNet  MATH  Google Scholar 

  30. Kuncheva LI, Jain LC (2000) Designing classifier fusion systems by genetic algorithms. IEEE Trans Evol Comput 4(4):327–336

    Article  Google Scholar 

  31. Lazar C, Taminau J, Meganck S, Steenhoff D, Coletta A, Molter C, Schaetzen V, Duque R, Bersini H, Nowe A (2012) A survey on filter techniques for feature selection in gene expression microarray analysis. IEEE ACM Trans Comput Biol Bioinform 9(4):1106–1119

    Article  Google Scholar 

  32. Laura EA, Santana A, Canuto MP (2012) Bi-objective genetic algorithm for feature selection in ensemble systems. In: Artificial neural networks and machine learning—ICANN 2012. LNCS, vol 7552. Springer, Berlin, pp 701–709

  33. Lehmann EL, Romano JP (2006) Testing statistical hypothese, vol 64, no 2. Springer, Berlin, pp 255–256

  34. Ma X, Huo J, Wang Q (2010) A multi-objective genetic algorithm approach based on the uniform design metmod. In: International conference on computational intelligence and security, Nanning, pp 160–164. https://doi.org/10.1109/cis.2010.43

  35. Maaranen H, Miettinen K, Makela MM (2004) A quasi-random initial population for genetic algorithms. Comput Math Appl 47(12):1885–1895

    Article  MathSciNet  MATH  Google Scholar 

  36. Mitra P, Murthy CA, Pal SK (2002) Unsupervised feature selection using feature similarity. IEEE Trans Pattern Anal Mach Intell 24(3):301–312

    Article  Google Scholar 

  37. Neumann JV (1996) Cellular automata. In: Burks AW (ed) Theory of self-reproducing automata. Chap. 2, University of Illinois Press, Champaign

  38. Oliveira LS, Sabourin R, Bortolozzi F, Suen CY (2003) Feature selection for ensembles: a hierarchical multi-objective genetic algorithm approach. In: Seventh international conference on document analysis and recognition-ICDAR, vol 2. IEEE Computer Society, Washington, p 676

  39. Pawlak Z (1997) Rough set approach to knowledge-based decision support. Eur J Oper Res 99(1):48–57

    Article  MATH  Google Scholar 

  40. Price K, Storn RM, Lampinen JA (2005) Differential evolution: a practical approach to global optimization. Natural computing series. Springer, New York

    MATH  Google Scholar 

  41. Rokach L (2010) Ensemble based classifiers. Artif Intell Rev 33(1–2):1–39

    Article  Google Scholar 

  42. Roth V, Lange T (2004) Bayesian class discovery in microarray dataset. IEEE Trans Biomed Eng 51(5):707–718

    Article  Google Scholar 

  43. Santana LS, Canuto AM (2014) Filter-based optimization techniques for selection of feature subsets in ensemble systems. Expert Syst Appl 41(4):1622–1631

    Article  Google Scholar 

  44. Schapire RE, Freund Y, Bartlett P (1998) Boosting the margin: a new explanation for the effectiveness of voting method. Ann Stat 26(5):1651–1686

    Article  MathSciNet  MATH  Google Scholar 

  45. Schölkopf AJ, Smola R, Bartlett P (2000) New support vector algorithms. Neural Comput 12(5):1207–1245

    Article  Google Scholar 

  46. Stoorvogel AA, Saberi A (2014) On global external stochastic stabilization of linear systems with input saturation. In: American control conference, OR, pp 2972–2976. https://doi.org/10.1109/acc.2014.6858588

  47. Teli S, Kanikar P (2015) A survey on decision tree based approaches in data mining. Int J Adv Res Comput Sci Softw Eng 5(4):613–617

    Google Scholar 

  48. Thandar AM, Khaing MK (2012) Radial basis function (RBF) neural network classification based on consistency evaluation measure. Int J Comput Appl 54(15):20–23

    Google Scholar 

  49. Trivedi A, Srinivasan D, Sanyal K, Ghosh A (2017) A survey of multiobjective evolutionary algorithms based on decomposition. IEEE Trans Evol Comput 21(3):440–462. https://doi.org/10.1109/TEVC.2016.2608507

    Article  Google Scholar 

  50. Wang H, Jiao L, Yao X (2014) An improved two-archive algorithm for many-objective optimization. IEEE Trans Evolut Comput. https://doi.org/10.1109/TEVC.2014.2350987

    Article  Google Scholar 

  51. Webb G, Zheng Z (2004) Multi-strategy ensemble learning: reducing error by combining ensemble learning techniques. IEEE Trans Knowl Data Eng 16(8):980–991

    Article  Google Scholar 

  52. Yang Y, Pedersen JO (1997) A comparative study on feature selection in text categorization. In: ICML, vol 97, pp 412–420

  53. Yang P, Zhang Z (2007) Hybrid methods to select informative gene sets in microarray data classification. In: Proceedings of AI 2007. LNAI, vol 4830. Springer, Berlin, pp 811–815

  54. Zhang Q, Li H (2007) MOEA/D: a multi-objective evolutionary algorithm based on decomposition. IEEE Trans Evolut Comput 11(6):712–731

    Article  Google Scholar 

  55. Zhang Z, Yang P (2008) An ensemble of classifiers with genetic algorithm based feature selection. IEEE Intell Inform Bull 9(1):18–24

    Google Scholar 

  56. Zitzler E, Thiele L (1999) Multi-objective evolutionary algorithms: a comparative case study and the strength Pareto approach. IEEE Trans Evol Comput 3(4):257–271

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank anonymous reviewers for their valuable comments.

Author information

Authors and Affiliations

  1. Computer Science and Technology, Indian Institute of Engineering Science and Technology, Shibpur, Howrah, West Bengal, 711103, India

    Asit Kumar Das & Arka Ghosh

  2. Bioinformatics, Maulana Abul Kalam Azad University of Technology, Haringhata, Nadia, West Bengal, 741249, India

    Soumen Kumar Pati

Authors
  1. Asit Kumar Das
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Soumen Kumar Pati
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Arka Ghosh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Asit Kumar Das.

Ethics declarations

Conflict of interest

The authors declare that there are no conflicts of interest in this paper.

Ethical approval

This article does not contain any studies with human participants performed by any of the authors.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Das, A.K., Pati, S.K. & Ghosh, A. Relevant feature selection and ensemble classifier design using bi-objective genetic algorithm. Knowl Inf Syst 62, 423–455 (2020). https://doi.org/10.1007/s10115-019-01341-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10115-019-01341-6

Keywords

Search

Navigation