ChartFuse: a novel fusion method for chart classification using heterogeneous microstructures | Multimedia Tools and Applications Skip to main content
Springer Nature Link
Log in

ChartFuse: a novel fusion method for chart classification using heterogeneous microstructures

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Chart images exhibit significant structural variabilities, which is called as micro-structural variabilities, which makes each image type different from others even though chart image belongs to the same class or categories. The lack of affiliation between the heterogeneous features and the structure of the chart images, make it challenging to learn these micro-variabilities features by any learning model for automatic chart recognition and interpretation. However, extracting low-level heterogeneous features from chart images remains challenging. This paper presents a novel chart image classification method by using local feature descriptor. We proposed a new heterogeneous feature extractor, namely the heterogeneity index (HI) fused with local penta pattern. Here, the microstructural features are defined on the similarity of the chroma effects, and HI is computed depending upon the basic colors and its intensity in the microstructures with similar chroma effects. HI integrates colors, textures, structural layout, and illumination details with the local features altogether for the image classification. The proposed method is tested on chart image datasets, namely FigureQA and our handcrafted chart dataset. Experimental results depict that our method classify images with an accuracy rate of 95%–97% which is an increase of 5%–10% as compared with the customary methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Agarwal M, Singhal A, Lall B (2019) Multi-channel local ternary pattern for content-based image retrieval. Pattern Anal Applic 22(4):1585–1596

  2. Ahmad J, Sajjad M, Mehmood I, Rho S, Baik SW (2015) Describing colors, textures and shapes for content based image retrieval-a survey. arXiv preprint arXiv:1502.07041

  3. Al-Ariny Z, Abdelwahab MA, Fakhry M, Hasaneen E-S (2020) An Efficient Vehicle Counting Method Using Mask R-CNN. In: 2020 International Conference on Innovative Trends in Communication and Computer Engineering (ITCE), IEEE, pp. 232–237

  4. Amara J, Kaur P, Owonibi M, Bouaziz B (2017) Convolutional neural network based chart image classification. 25th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision in co-operation with EUROGRAPHICS Association, pp 83–88

  5. Ashish O, Varun B, Rohini S, Manpreet S (2013) A Framework for Medical Image Retrieval Using Local Tetra Pattern. Int J Eng Sci Technol 5:27

    Google Scholar 

  6. Bala A, Kaur T (2016) Local texton XOR patterns: a new feature descriptor for content-based image retrieval. Eng Sci Technol Int J 19(1):101–112

    Google Scholar 

  7. Chahi A, Ruichek Y, Touahni R (2018) Local directional ternary pattern: a new texture descriptor for texture classification. Comput Vis Image Underst 169:14–27

    Article  Google Scholar 

  8. Chakraborty S, Singh SK, Chakraborty P (2017) Local quadruple pattern: a novel descriptor for facial image recognition and retrieval. Comput Electr Eng 62:92–104

    Article  Google Scholar 

  9. Chakraborty S, Singh SK, Chakraborty P (2019) Cascaded asymmetric local pattern: a novel descriptor for unconstrained facial image recognition and retrieval. Multimed Tools Appl 78(17):25143–25162

  10. Chen M, Golan A (2015) What may visualization processes optimize? IEEE Trans Vis Comput Graph 22(12):2619–2632

    Article  Google Scholar 

  11. Choi J, Jung S, Park DG, Choo J, Elmqvist N (2019) Visualizing for the non-visual: enabling the visually impaired to use visualization. In: Computer Graphics Forum, Vol. 38, Wiley Online Library, pp. 249{260

  12. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and pattern recognition, CVPR'05, Vol. 1, pp 886–893

  13. Dandashy T, El Hassan M, Bitar A (2019) Enhanced face detection based on haar-like and MB-LBP features. Int J Eng Manag Res. https://doi.org/10.31033/ijemr.9.4.17

  14. Davila K, Setlur S, Doermann D, Bhargava UK, Govindaraju V (2020) Chart Mining: a survey of methods for automated chart analysis. IEEE Trans Patt Anal Mach Intell. https://doi.org/10.1109/TPAMI.2020.2992028

  15. Dubey SR, Singh SK, Singh RK (2016) Multichannel decoded local binary patterns for content-based image retrieval. IEEE Trans Image Process 25(9):4018–4032

    Article  MathSciNet  Google Scholar 

  16. Fadaei S, Amirfattahi R, Ahmadzadeh MR (2017) Local derivative radial patterns: a new texture descriptor for content-based image retrieval. Signal Process 137:274–286

    Article  Google Scholar 

  17. Fu X, Wei W (2008) Centralized binary patterns embedded with image Euclidean distance for facial expression recognition. In: 2008 Fourth International Conference on Natural Computation, vol. 4, pp. 115–119. IEEE

  18. Ganesh P, Volle K, Burks TF, Mehta SS (2019 Jan 1) Deep Orange: mask R-CNN based Orange detection and segmentation. IFAC-PapersOnLine. 52(30):70–75

    Article  MathSciNet  Google Scholar 

  19. Hao Q, Feng Q, Wei Y, Sbert M, Lu W, Qing X (2018) Pairwise cross pattern: a color-LBP descriptor for content-based image retrieval. In: Pacific Rim Conference on Multimedia, pp. 290–300. Springer, Cham

  20. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778

  21. Höschl C IV, Flusser J (2016) Robust histogram-based image retrieval. Pattern Recogn Lett 69:72–81

    Article  Google Scholar 

  22. Jung D, et al. (2017) ChartSense: interactive data extraction from chart images." Proceedings of the 2017 CHI Conference on Human Fac-tors in Computing Systems. ACM

  23. Kahou SE, Michalski V, Atkinson A, Kádár Á, Trischler A, Bengio Y (2017) Figureqa: an annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300

  24. Karthikeyani V, Nagarajan S (2012) Machine learning classification algorithms to recognize chart types in portable document format (pdf) files. Int J Comput Appl 39:1–5

    Google Scholar 

  25. Kou Q, Cheng D, Chen L, Zhao K (2018) A multiresolution gray-scale and rotation invariant descriptor for texture classification. IEEE Access 6:30691–30701

    Article  Google Scholar 

  26. Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Proceedings of the 25th International Conference on Neural Information Processing Systems -Volume 1 (NIPS'12), Pereira F, Burges CJC, Bottou L, Weinberger KQ (Eds.), Vol. 1. Curran Associates Inc., USA, pp. 1097–1105.

  27. Krizhevsky A, Sutskever I, Hinton G (2012) ImageNet classification with deep convolutional networks. Neur Info Process Syst 1:1097–1105

    Google Scholar 

  28. Kumar TGS, Nagarajan V (2018) Local contourlet tetra pattern for image retrieval. SIViP 12(3):591–598

    Article  Google Scholar 

  29. Kumar K, Li J-P, Khan I (2016) A comparative study among colorful image descriptors for content based image retrieval. In: 2016 3rd International Conference on Computing for Sustainable Global Development (INDIACom), pp. 3922–3926. IEEE

  30. Lan R, Zhong S, Liu Z, Shi Z, Luo X (2018) A simple texture feature for retrieval of medical images. Multimed Tools Appl 77(9):10853–10866

    Article  Google Scholar 

  31. Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324. https://doi.org/10.1109/5.726791

    Article  Google Scholar 

  32. Liao S, Zhu X, Lei Z, Zhang L, Li SZ (2007) Learning multi-scale block local binary patterns for face recognition. In International Conference on Biometrics, pp. 828–837. Springer, Berlin, Heidelberg

  33. Liu P, Guo J-M, Chamnongthai K, Prasetyo H (2017) Fusion of color histogram and LBP-based features for texture image retrieval and classification. Inf Sci 390:95–111

    Article  Google Scholar 

  34. Lowe DG (1999) Object recognition from local scale-invariant features. Iccv 99(2):1150–1157

    Google Scholar 

  35. Murala S, Maheshwari RP, Balasubramanian R (2012) Local tetra patterns: a new feature descriptor for content-based image retrieval. IEEE Trans Image Process 21(5):2874–2886

    Article  MathSciNet  Google Scholar 

  36. Ojala T, Pietikäinen M, Harwood D (1996) A comparative study of texture measures with classification based on featured distributions. Pattern Recogn 29(1):51–59

    Article  Google Scholar 

  37. Oliva A, Torralba A (2001) Modeling the shape of the scene: a holistic representation of the spatial envelope. Int J Comput Vis 42(3):145–175

    Article  Google Scholar 

  38. Poco J, Heer J (2017) Reverse-engineering visualizations: recovering visual encodings from chart images. Comp Graph Forum 36(3):353–363

  39. Pouyanfar S, Sadiq S, Yan Y, Tian H, Tao Y, Reyes MP, Shyu M-L, Chen S-C, Iyengar SS (2018) A survey on deep learning: algorithms, techniques, and applications. ACM Comput Surveys (CSUR) 51(5):1–36

    Article  Google Scholar 

  40. Prasad VSN, Siddiquie B, Golbeck J, Davis LS (2007) Classifying computer generated charts. In: Content Based Multimedia Indexing. CBMI'07. International Workshop on. IEEE, 85 92

  41. Savva M, Kong N, Chhajta A, Fei Fei L, Agrawala M, Heer J (2011) ReVision: automated classification, analysis and redesign of chart images. In: Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST '11). ACM, New York, NY, USA, 393 402.

  42. Shakoor MH, Boostani R (2017) Extended mapping local binary pattern operator for texture classification. Int J Pattern Recognit Artif Intell 31(06):1750019

    Article  Google Scholar 

  43. Simonyan K, Zisserman A (2019) Very deep convolutional networks for large-scale image recognition. Sep. 4, 2014. arXiv preprint arXiv:1409.1556

  44. Singh C, Walia E, Kaur KP (2018) Color texture description with novel local binary patterns for effective image retrieval. Pattern Recogn 76:50–68

    Article  Google Scholar 

  45. Suhasini PS, Krishna KSR, Krishna M IV (2017) Content based image retrieval based on different global and local color histogram methods: a survey. J Inst Eng (India): Series B 98(1):129–135

    Google Scholar 

  46. Tan X, Triggs B (2007) Enhanced local texture feature sets for face recognition under difficult lighting conditions. In: International workshop on analysis and modeling of faces and gestures, Springer, Berlin, Heidelberg, pp. 168–182

  47. Tan C, Sun F, Kong T, Zhang W, Yang C, Liu C (2018) A survey on deep transfer learning. In: International conference on artificial neural networks. Springer, Cham, pp. 270–279

  48. Tang B, Liu X, Lei J, Song M, Tao D, Sun S, Dong F (2016) Deepchart: combining deep convolutional net-works and deep belief networks in chart classification. Signal Process 124:156–161

    Article  Google Scholar 

  49. Tao G, Feng XL, Chen F, Zhai JH (2013) Local comprehensive patterns: a novel face feature descriptor. Optik 124(24):7022–7026

    Article  Google Scholar 

  50. Verma M, Raman B (2018) Local neighborhood difference pattern: a new feature descriptor for natural and texture image retrieval. Multimed Tools Appl 77(10):11843–11866

    Article  Google Scholar 

  51. Vipparthi SK, Nagar SK (2014) Color directional local quinary patterns for content based indexing and retrieval. Hum-Centric Comput Info Sci 4(1):6

    Article  Google Scholar 

  52. Youssef N, Algergawy A, Moawad IF, EL-Horbaty EL-SM (2018) Combined features for content based image retrieval: a comparative study. In: International Conference on Advanced Intelligent Systems and Informatics, Springer, Cham, pp. 634–643

  53. Zhang B, Gao Y, Zhao S, Liu J (2009) Local derivative pattern versus local binary pattern: face recognition with high-order local pattern descriptor. IEEE Trans Image Process 19(2):533–544

    Article  MathSciNet  Google Scholar 

  54. Zhang B, Zhang L, Zhang D, Shen L (2010) Directional binary code with application to PolyU near-infrared face database. Pattern Recogn Lett 31(14):2337–2344

    Article  Google Scholar 

  55. Zhao Z, Tian Q, Sun H, Jin X, Guo J (2016) Content based image retrieval scheme using color, texture and shape features. Int J Signal Process Image Process Patt Recognit 9(1):203–212

    Google Scholar 

  56. Zhou YP, Tan CL (2001) Learning-based scientific chart recognition. In 4th IAPR International Workshop on Graphics Recognition, GREC, pp. 482-492

  57. Zhou W, Li H, Tian Q (2017) Recent advance in content-based image retrieval: a literature survey. arXiv preprint arXiv:1706.06064

  58. Zhou J-X, Liu X-d, Xu T-W, Gan J, Liu W (2018) A new fusion approach for content based image retrieval with color histogram and local directional pattern. Int J Mach Learn Cybern 9(4):677–689

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, DSPM-IIITNR, Raipur, India

    Prerna Mishra & Santosh Kumar

  2. Deparment of Mathematics, DSPM-IIITNR, Raipur, India

    Mithilesh Kumar Chaube

Authors
  1. Prerna Mishra
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Santosh Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mithilesh Kumar Chaube
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Prerna Mishra.

Ethics declarations

Conflict of interest

Authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mishra, P., Kumar, S. & Chaube, M.K. ChartFuse: a novel fusion method for chart classification using heterogeneous microstructures. Multimed Tools Appl 80, 10417–10439 (2021). https://doi.org/10.1007/s11042-020-10186-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-10186-z

Keywords

Search

Navigation