Efficient GitHub Crawling Using the GraphQL API | SpringerLink
Skip to main content
Springer Nature Link
Log in

Efficient GitHub Crawling Using the GraphQL API

  • Conference paper
  • First Online:
Computational Science and Its Applications – ICCSA 2022 Workshops (ICCSA 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13381))

Included in the following conference series:

Abstract

The number of publicly accessible software repositories on online platforms is growing rapidly. With more than 128 million public repositories (as of March 2020), GitHub is the world’s largest platform for hosting and managing software projects. Where it used to be necessary to merge various data sources, it is now possible to access a wealth of data using the GitHub API alone. However, collecting and analyzing this data is not an easy endeavor. In this paper, we present Prometheus, a system for crawling and storing software repositories from GitHub. Compared to existing frameworks, Prometheus follows an event-driven microservice architecture. By separating functionality on the service level, there is no need to understand implementation details or use existing frameworks to extend or customize the system, only data. Prometheus consists of two components, one for fetching GitHub data and one for data storage which serves as a basis for future functionality. Unlike most existing crawling approaches, the Prometheus fetching service uses the GitHub GraphQL API. As a result, Prometheus can significantly outperform alternatives in terms of throughput in some scenarios.

We want to thank the anonymous reviewers for their valuable feedback to improve this article. This work is part of the “Software-DNA” project, which is funded by the European Regional Development Fund (ERDF or EFRE in German) and the State of Brandenburg (ILB). This work is part of the KMU project “KnowhowAnalyzer” (Förderkennzeichen 01IS20088B), which is funded by the German Ministry for Education and Research (Bundesministerium für Bildung und Forschung).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 13727
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 17159
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Bjertnes, L., Tørring, J.O., Elster, A.C.: LS-CAT: a large-scale CUDA AutoTuning dataset. In: 2021 International Conference on Applied Artificial Intelligence (ICAPAI), pp. 1–6. IEEE (2021). https://doi.org/10.1109/ICAPAI49758.2021.9462050

  2. Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)

    MATH  Google Scholar 

  3. Borges, H., Hora, A., Valente, M.T.: Understanding the factors that impact the popularity of github repositories. In: 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 334–344 (2016). https://doi.org/10.1109/ICSME.2016.31

  4. Brito, G., Mombach, T., Valente, M.T.: Migrating to GraphQL: a practical assessment. In: Proceedings of 26th International Conference on Software Analysis, Evolution and Reengineering, SANER 2019, pp. 140–150. IEEE (2019). https://doi.org/10.1109/SANER.2019.8667986

  5. Brito, G., Valente, M.T.: REST vs GraphQL: a controlled experiment. In: Proceedings of International Conference on Software Architecture, ICSA 2020, pp. 81–91. IEEE (2020). https://doi.org/10.1109/ICSA47634.2020.00016

  6. di Cosmo, R., Zacchiroli, S.: Software heritage: why and how to preserve software source code. In: iPRES 2017–14th International Conference on Digital Preservation, pp. 1–10 (2017)

    Google Scholar 

  7. Dyer, R., Nguyen, H.A., Rajan, H., Nguyen, T.N.: Boa: Ultra-large-scale software repository and source-code mining. ACM Trans. Softw. Eng. Methodol. 25(1), 1–34 (2015). https://doi.org/10.1145/2803171

    Article  Google Scholar 

  8. de F. Farias, M.A., Novais, R., Júnior, M.C., da Silva Carvalho, L.P., Mendonça, M., Spínola, R.O.: A systematic mapping study on mining software repositories. In: Proceedings of the 31st Annual ACM Symposium on Applied Computing, SAC 2016, pp. 1472–1479. ACM (2016). https://doi.org/10.1145/2851613.2851786

  9. Fowler, M.: Event sourcing (2005). https://martinfowler.com/eaaDev/EventSourcing.html. Accessed 17 May 2022

  10. Fowler, M., Lewis, J.: Microservices (2014). https://www.martinfowler.com/articles/microservices.html. Accessed 17 May 2022

  11. Fowler, M.: What do you mean by “Event-Driven”? (2017). https://martinfowler.com/articles/201701-event-driven.html. Accessed 17 May 2022

  12. Gasparini, M., Clarisó, R., Brambilla, M., Cabot, J.: Participation inequality and the 90-9-1 principle in open source. In: Proceedings of the 16th International Symposium on Open Collaboration, pp. 1–7. ACM (2020). https://doi.org/10.1145/3412569.3412582

  13. Github: List repository issues. https://docs.github.com/en/rest/reference/issues#list-repository-issues. Accessed 17 May 2022

  14. Gousios, G., Spinellis, D.: GHTorrent: Github’s data from a firehose. In: Proceedings of 9th International Workshop on Mining Software Repositories, MSR 2012, pp. 12–21. IEEE/ACM (2012). https://doi.org/10.1109/MSR.2012.6224294

  15. Gousios, G.: The GHTorrent dataset and tool suite. In: Proceedings of the 10th Working Conference on Mining Software Repositories, MSR 2013, pp. 233–236. IEEE (2013). https://doi.org/10.1109/MSR.2013.6624034

  16. Gousios, G., Vasilescu, B., Serebrenik, A., Zaidman, A.: Lean GHTorrent: GitHub data on demand. In: Proceedings of the 11th Working Conference on Mining Software Repositories, MSR 2014, pp. 384–387. ACM (2014). https://doi.org/10.1145/2597073.2597126

  17. Hagiwara, M., Mita, M.: Github typo corpus: a large-scale multilingual dataset of misspellings and grammatical errors. arXiv preprint arXiv:1911.12893 (2019)

  18. Hartig, O., Pérez, J.: Semantics and complexity of GraphQL. In: Proceedings of World Wide Web Conference, WWW 2018, pp. 1155–1164. International World Wide Web Conferences Steering Committee (2018). https://doi.org/10.1145/3178876.3186014

  19. Jaramillo, D., Nguyen, D.V., Smart, R.: Leveraging microservices architecture by using docker technology. In: SoutheastCon 2016, pp. 1–5 (2016). https://doi.org/10.1109/SECON.2016.7506647

  20. Kalliamvakou, E., Gousios, G., Blincoe, K., Singer, L., German, D.M., Damian, D.: The promises and perils of mining github. In: Proceedings of the 11th Working Conference on Mining Software Repositories, MSR 2014, pp. 92–101. ACM (2014). https://doi.org/10.1145/2597073.2597074

  21. Leberknight, S.: Polyglot persistence (2008). http://www.sleberknight.com/blog/sleberkn/entry/polyglot_persistence. Accessed 17 May 2022

  22. Linstead, E., Bajracharya, S., Ngo, T., Rigor, P., Lopes, C., Baldi, P.: Sourcerer: mining and searching internet-scale software repositories. Data Min. Knowl. Disc. 18(2), 300–336 (2009). https://doi.org/10.1007/s10618-008-0118-x

    Article  MathSciNet  Google Scholar 

  23. Ma, Y., Bogart, C., Amreen, S., Zaretzki, R., Mockus, A.: World of code: an infrastructure for mining the universe of open source VCS data. In: Proceedings of 16th International Workshop on Mining Software Repositories, MSR 2019, pp. 143–154. IEEE/ACM (2019). https://doi.org/10.1109/MSR.2019.00031

  24. Mavroudeas, G., et al.: Learning GraphQL query cost. In: Proceedings of 36th International Conference on Automated Software Engineering, ASE 2021, pp. 1146–1150. IEEE/ACM (2021). https://doi.org/10.1109/ASE51524.2021.9678513

  25. Mei, S.: Why you should never use mongodb (2013). http://www.sarahmei.com/blog/2013/11/11/why-you-should-never-use-mongodb. Accessed 17 May 2022

  26. Menzies, T., Zimmermann, T.: Software analytics: so what? IEEE Softw. 30, 31–37 (2013). https://doi.org/10.1109/MS.2013.86

    Article  Google Scholar 

  27. Munaiah, N., Kroh, S., Cabrey, C., Nagappan, M.: Curating GitHub for engineered software projects. Empir. Softw. Eng. 22(6), 3219–3253 (2017). https://doi.org/10.1007/s10664-017-9512-6

    Article  Google Scholar 

  28. Ortu, M., Destefanis, G., Adams, B., Murgia, A., Marchesi, M., Tonelli, R.: The JIRA repository dataset: understanding social aspects of software development. In: Proceedings of the 11th International Conference on Predictive Models and Data Analytics in Software Engineering, PROMISE 2015, pp. 1–4. ACM (2015). https://doi.org/10.1145/2810146.2810147

  29. Rosen-Zvi, M., Griffiths, T., Steyvers, M., Smyth, P.: The author-topic model for authors and documents. In: Proceedings of 20th Conference on Uncertainty in Artificial Intelligence, UAI 2004, pp. 487–494. AUAI Press (2004)

    Google Scholar 

  30. Seabra, M., Nazário, M.F., Pinto, G.: REST or GraphQL? A performance comparative study. In: Proceedings of XIII Brazilian Symposium on Software Components, Architectures, and Reuse, SBCARS 2019, pp. 123–132. ACM (2019). https://doi.org/10.1145/3357141.3357149

  31. Tiwari, N.M., Upadhyaya, G., Rajan, H.: Candoia: a platform and ecosystem for mining software repositories tools. In: 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C), pp. 759–761 (2016)

    Google Scholar 

  32. Trautsch, A., Trautsch, F., Herbold, S., Ledel, B., Grabowski, J.: The SmartSHARK ecosystem for software repository mining. In: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Companion Proceedings, pp. 25–28. ACM (2020). https://doi.org/10.1145/3377812.3382139

  33. Wittern, E., Cha, A., Davis, J.C., Baudart, G., Mandel, L.: An empirical study of GraphQL schemas. In: Yangui, S., Bouassida Rodriguez, I., Drira, K., Tari, Z. (eds.) ICSOC 2019. LNCS, vol. 11895, pp. 3–19. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-33702-5_1

    Chapter  Google Scholar 

  34. Zhang, D., Han, S., Dang, Y., Lou, J.G., Zhang, H., Xie, T.: Software analytics in practice. IEEE Softw. 30, 30–37 (2013). https://doi.org/10.1109/MS.2013.94

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Hasso Plattner Institute, Digital Engineering Faculty, University of Potsdam, Potsdam, Germany

    Adrian Jobst, Daniel Atzberger, Tim Cech, Willy Scheibel, Matthias Trapp & Jürgen Döllner

Authors
  1. Adrian Jobst
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniel Atzberger
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tim Cech
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Willy Scheibel
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Matthias Trapp
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jürgen Döllner
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Adrian Jobst .

Editor information

Editors and Affiliations

  1. University of Perugia, Perugia, Italy

    Osvaldo Gervasi

  2. University of Basilicata, Potenza, Potenza, Italy

    Beniamino Murgante

  3. Østfold University College, Halden, Norway

    Sanjay Misra

  4. University of Minho, Braga, Portugal

    Ana Maria A. C. Rocha

  5. University of Cagliari, Cagliari, Italy

    Chiara Garau

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jobst, A., Atzberger, D., Cech, T., Scheibel, W., Trapp, M., Döllner, J. (2022). Efficient GitHub Crawling Using the GraphQL API. In: Gervasi, O., Murgante, B., Misra, S., Rocha, A.M.A.C., Garau, C. (eds) Computational Science and Its Applications – ICCSA 2022 Workshops. ICCSA 2022. Lecture Notes in Computer Science, vol 13381. Springer, Cham. https://doi.org/10.1007/978-3-031-10548-7_48

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-10548-7_48

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-10547-0

  • Online ISBN: 978-3-031-10548-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation