Abstract
There’s a common agreement in the industry that integration and end-to-end (e2e) tests are a challenge for many teams wanting to enable frequent deployments while at the same time guaranteeing quality. What this means for the research community is that there are open research problems that might be interesting to solve. However, little effort is put by academia on these integration and e2e tests. Truth is that all datasets available for research in software testing are focused on unit tests.
In this paper we propose an approach to build datasets of e2e tests from active open source projects. The approach is based on mining open source repositories from GitHub, in order to find those projects containing e2e tests. We defined 12 different criteria to find those tests. We investigate which of the 12 criteria are more reliable for detecting this kind of tests by manually analyzing the results of these criteria on 100 projects from GitHub. Then we performed a search on 1,800 projects (900 Java-specific, and 900 not constrained to Java), and used the three most promising criteria to detect e2e tests within all of them. Our results show that it is easier to detect this kind of tests in Java projects than on projects using other programming languages. Also, more than 500 projects were reported as having e2e tests. We hypothesize that good e2e test datasets could be built out of these results.
This work has been supported by the Government of Spain through project “BugBirth” (RTI2018-101963-B-100), the Regional Government of Madrid (CM) through project Cloud4BigData (S2013/ICE-2894) cofunded by FSE & FEDER and the European Commission through European Project H2020 822717: MICADO.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Biswas, S., Islam, M.J., Huang, Y., Rajan, H.: Boa meets python: a boa dataset of data science software in python language. In: 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR), pp. 577–581. IEEE (2019)
Dallmeier, V., Zimmermann, T.: Extraction of bug localization benchmarks from history. In: Proceedings of the Twenty-second IEEE/ACM International Conference on Automated Software Engineering, ASE 2007, pp. 433–436. ACM, New York (2007). https://doi.org/10.1145/1321631.1321702. http://doi.acm.org/10.1145/1321631.1321702
Gyimesi, P., Vancsics, B., Stocco, A., Mazinanian, D., Árpád Beszédes, Ferenc, R., Mesbah, A.: BugsJS: a benchmark of javascript bugs. In: Proceedings of 12th IEEE International Conference on Software Testing, Verification and Validation (ICST) (2019)
Just, R., Jalali, D., Ernst, M.D.: Defects4J: a database of existing faults to enable controlled testing studies for java programs. In: Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA 2014, pp. 437–440. ACM, New York (2014). https://doi.org/10.1145/2610384.2628055. http://doi.acm.org/10.1145/2610384.2628055
Karampatsis, R.M., Sutton, C.: How often do single-statement bugs occur? The manysstubs4j dataset. In: Proceedings of the 17th International Conference on Mining Software Repositories, pp. 573–577 (2020)
Memon, A., et al.: Taming google-scale continuous testing. In: Proceedings of the 39th International Conference on Software Engineering: Software Engineering in Practice Track, ICSE-SEIP 2017, pp. 233–242. IEEE Press, Piscataway (2017). https://doi.org/10.1109/ICSE-SEIP.2017.16
Saha, R., Lyu, Y., Lam, W., Yoshida, H., Prasad, M.: Bugs. jar: a large-scale, diverse dataset of real-world java bugs. In: 2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR), pp. 10–13. IEEE (2018)
Soto-Sánchez, Ó., Maes-Bermejo, M., Gallego, M., Gortázar, F.: A dataset of regressions in web applications detected by end-to-end tests. In: Shepperd, M., Brito e Abreu, F., Rodrigues da Silva, A., Pérez-Castillo, R. (eds.) QUATIC 2020. CCIS, vol. 1266, pp. 439–448. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58793-2_35
Widyasari, R., et al.: BugsInPy: a database of existing bugs in python programs to enable controlled testing and debugging studies. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020, pp. 1556–1560. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3368089.3417943
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Gortázar, F., Maes-Bermejo, M., Gallego, M., Contreras Padilla, J. (2021). Looking for the Needle in the Haystack: End-to-end Tests in Open Source Projects. In: Paiva, A.C.R., Cavalli, A.R., Ventura Martins, P., Pérez-Castillo, R. (eds) Quality of Information and Communications Technology. QUATIC 2021. Communications in Computer and Information Science, vol 1439. Springer, Cham. https://doi.org/10.1007/978-3-030-85347-1_4
Download citation
DOI: https://doi.org/10.1007/978-3-030-85347-1_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-85346-4
Online ISBN: 978-3-030-85347-1
eBook Packages: Computer ScienceComputer Science (R0)