What Will You Tell Me About the Chart? – Automated Description of Charts | SpringerLink
Skip to main content
Springer Nature Link
Log in

What Will You Tell Me About the Chart? – Automated Description of Charts

  • Conference paper
  • First Online:
Neural Information Processing (ICONIP 2021)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1516))

Included in the following conference series:

Abstract

An automatic chart description is a very challenging task. There are many more relationships between objects in a chart compared to general computer vision problems. Furthermore, charts have a different specificity to natural-scene pictures, so commonly used methods do not perform well. To tackle these problems, we propose a process consisting of three sub-tasks: (1) chart classification, (2) detection of a chart’s essential elements, and (3) generation of text description.

Due to the lack of plot datasets dedicated to the task of generating text, we prepared a new dataset – ChaTa+ which contains real-made figures. Additionally, we have adjusted publicly available FigureQA and PlotQA datasets to our particular tasks and tested our method on them. We compared our results with those of the Adobe team [3], which we treated as a benchmark. Finally, we obtained comparable results of the models’ performance, although we trained them on a more complex dataset (semi-synthetic PlotQA) and built a less resource-intensive infrastructure.

Research was funded by the Centre for Priority Research Area Artificial Intelligence and Robotics of Warsaw University of Technology within the Excellence Initiative: Research University (IDUB) programme (grant no 1820/27/Z01/POB2/2021).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 13727
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 17159
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://github.com/mini-pw/2019L-ProjektZespolowy.

  2. 2.

    https://github.com/EagleW/Describing_a_Knowledge_Base.

  3. 3.

    https://github.com/grant-TraDA.

References

  1. Bahdanau, D.: Neural machine translation by jointly learning to align and translate. In: ICLR (2015)

    Google Scholar 

  2. Behzadian, M., Otaghsara, S., Yazdani, M., Ignatius, J.: A state-of the-art survey of TOPSIS applications. Expert Syst. Appl. 39, 13051–13069 (2012)

    Article  Google Scholar 

  3. Chen, C., Zhang, R., et al.: Figure captioning with relation maps for reasoning. In: WACV (2020)

    Google Scholar 

  4. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2015)

    Google Scholar 

  5. Jobin, K.V., Mondal, A., Jawahar, C.V.: Docfigure: a dataset for scientific document figure classification. In: ICDAR (2019)

    Google Scholar 

  6. Kafle, K., Price, B., Cohen, S., Kanan, C.: DVQA: understanding data visualizations via question answering. In: CVPR (2018)

    Google Scholar 

  7. Kahou, S.E., Michalski, V., Atkinson, A., Kadar, A., Trischler, A., Bengio, Y.: FigureQA: an annotated figure dataset for visual reasoning. In: ICLR (2018)

    Google Scholar 

  8. Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: CVPR (2017)

    Google Scholar 

  9. Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: ICCV (2017)

    Google Scholar 

  10. Lin, T.Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  11. Nitesh, M., Pritha, G., Mitesh, K., Pratyush, K.: PlotQA: reasoning over scientific plots. In: WACV (2020)

    Google Scholar 

  12. Ren, S., He, K., Girshick, R.B., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: NIPS (2015)

    Google Scholar 

  13. Savva, M., Kong, N., Chhajta, A., Fei-Fei, L., Agrawala, M., Heer, J.: ReVision: automated classification, analysis and redesign of chart images. In: ACM (2011)

    Google Scholar 

  14. Siegel, N., Horvitz, Z., Levin, R., Divvala, S., Farhadi, A.: FigureSeer: parsing result-figures in research papers. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 664–680. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46478-7_41

    Chapter  Google Scholar 

Download references

Acknowledgements

We would like to thank to Przemysław Biecek and Tomasz Stanisławek for their work on common idea for creating the ChaTa dataset of Charts and Tables along with annotations of their elements, and preliminary ideas of the system to annotate them, and we are grateful for many students from the Faculty of Mathematics and Information Science who contributed to the annotation tool and gathering the preliminary ChaTa dataset, which we modified further.

Author information

Authors and Affiliations

  1. Faculty of Mathematics and Information Science, Warsaw University of Technology, Warsaw, Poland

    Karolina Seweryn, Katarzyna Lorenc & Anna Wróblewska

  2. Management Faculty, Warsaw University of Technology, Warsaw, Poland

    Sylwia Sysko-Romańczuk

Authors
  1. Karolina Seweryn
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Katarzyna Lorenc
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Anna Wróblewska
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sylwia Sysko-Romańczuk
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Karolina Seweryn or Anna Wróblewska .

Editor information

Editors and Affiliations

  1. Sampoerna University, Jakarta, Indonesia

    Teddy Mantoro

  2. Kyungpook National University, Daegu, Korea (Republic of)

    Minho Lee

  3. Sampoerna University, Jakarta, Indonesia

    Media Anugerah Ayu

  4. Murdoch University, Murdoch, WA, Australia

    Kok Wai Wong

  5. Universitas Indonesia, Depok, Indonesia

    Achmad Nizar Hidayanto

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Seweryn, K., Lorenc, K., Wróblewska, A., Sysko-Romańczuk, S. (2021). What Will You Tell Me About the Chart? – Automated Description of Charts. In: Mantoro, T., Lee, M., Ayu, M.A., Wong, K.W., Hidayanto, A.N. (eds) Neural Information Processing. ICONIP 2021. Communications in Computer and Information Science, vol 1516. Springer, Cham. https://doi.org/10.1007/978-3-030-92307-5_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-92307-5_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-92306-8

  • Online ISBN: 978-3-030-92307-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation