Abstract
Automated testing approaches for RESTful web APIs typically follow a black-box strategy, where test cases are derived from the API specification. These techniques show promising results, but they neglect constraints among input parameters (so-called inter-parameter dependencies), as these cannot be formally described in current API specification languages. As a result, black-box tools rely on brute force to generate valid test cases, i.e., those satisfying all the input constraints. This is not only extremely inefficient, but it is also unlikely to work for most real-world services, where inter-parameter dependencies are complex and pervasive. In this paper, we present RESTest, a framework for automated black-box testing of RESTful APIs. Among its key features, RESTest supports the specification and automated analysis of inter-parameter dependencies, enabling the use of constraint solvers for the automated generation of valid test cases. This allows to detect more faults, and faster, through a deeper evaluation of valid and invalid input parameters’ combinations and the use of novel test oracles. Evaluation results on 6 commercial APIs show that RESTest can efficiently generate up to 99% more valid test cases than random testing techniques, 60% on average. More importantly, RESTest revealed 2K failures undetected by random testing, uncovering bugs in all the services under test.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
This is reflected in an open feature request in OAS entitled “Support interdependencies between query parameters”, with over 290 votes and 55 comments from 33 participants. https://github.com/OAI/OpenAPI-Specification/issues/256.
- 2.
References
APIs.guru. https://apis.guru. Accessed Apr 2020
Arcuri, A.: RESTful API automated test case generation with EvoMaster. ACM TOSEM 28(1), 1–37 (2019)
Atlidakis, V., Godefroid, P., Polishchuk, M.: Checking security properties of cloud services REST APIs. In: ICST (2020)
Ed-douibi, H., Izquierdo, J.L.C., Cabot, J.: Automatic generation of test cases for REST APIs: a specification-based approach. In: EDOC, pp. 181–190 (2018)
Fielding, R.T.: Architectural styles and the design of network-based software architectures. Ph.D. thesis (2000)
Gamez-Diaz, A., Fernandez, P., Ruiz-Cortes, A.: Automating SLA-driven API development with SLA4OAI. In: ICSOC, pp. 20–35 (2019)
Gamez-Diaz, A., Fernandez, P., Ruiz-Cortés, A.: Governify for APIs: SLA-driven ecosystem for API governance. In: ESEC/FSE, pp. 1120–1123 (2019)
Gotlieb, A.: Constraint-based testing: an emerging trend in software testing. In: Advances in Computers, vol. 99, pp. 67–101. Elsevier (2015)
Karlsson, S., Causevic, A., Sundmark, D.: QuickREST: property-based test generation of OpenAPI described RESTful APIs. In: ICST (2020)
Li, Y., Sun, Z.A., Fang, J.Y.: Generating an automated test suite by variable strength combinatorial testing for web services. CIT 24(3), 271–282 (2016)
Martin-Lopez, A., Segura, S., Ruiz-Cortés, A.: A catalogue of inter-parameter dependencies in RESTful web APIs. In: ICSOC, pp. 399–414 (2019)
Martin-Lopez, A., Segura, S., Müller, C., Ruiz-Cortés, A.: Specification and automated analysis of inter-parameter dependencies in web APIs. IEEE Trans. Serv. Comput. (2020, Submitted to). https://bit.ly/2ECr9rc
Martin-Lopez, A., Segura, S., Ruiz-Cortés, A.: Test coverage criteria for RESTful web APIs. In: A-TEST, pp. 15–21 (2019)
MiniZinc: Constraint Modeling Language. https://www.minizinc.org. Accessed Apr 2020
OpenAPI Specification. https://www.openapis.org. Accessed Apr 2020
Semantic Markup for Web Services (OWL-S). https://www.w3.org/Submission/OWL-S. Accessed May 2020
REST Assured. http://rest-assured.io. Accessed Apr 2020
Richardson, L., Amundsen, M., Ruby, S.: RESTful Web APIs. O’Reilly Media Inc., Sebastopol (2013)
Segura, S., Parejo, J.A., Troya, J., Ruiz-Cortés, A.: Metamorphic testing of RESTful web APIs. IEEE TSE 44(11), 1083–1099 (2018)
Sun, C.a., Li, M., Jia, J., Han, J.: Constraint-based model-driven testing of web services for behavior conformance. In: ICSOC, pp. 543–559 (2018)
Supplementary material of the paper. https://github.com/isa-group/icsoc-2020-supplementary-material
Swagger. http://swagger.io. Accessed Apr 2020
Viglianisi, E., Dallago, M., Ceccato, M.: RestTestGen: automated black-box testing of RESTful APIs. In: ICST (2020)
Web Services Description Language (WSDL) Version 2.0. https://www.w3.org/TR/wsdl20. Accessed May 2020
Xu, L., Yuan, Q., Wu, J., Liu, C.: Ontology-based web service robustness test generation. In: WSE, pp. 59–68 (2009)
Acknowledgements
This work has been partially supported by the European Commission (FEDER) and Junta de Andalucia under projects APOLO (US-1264651) and EKIPMENT-PLUS (P18-FR-2895), by the Spanish Government under project HORATIO (RTI2018-101204-B-C21), and by the FPU scholarship program, granted by the Spanish Ministry of Education and Vocational Training (FPU17/04077). We would also like to thank Ramon Fernandez for his technical support during the development of RESTest.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Martin-Lopez, A., Segura, S., Ruiz-Cortés, A. (2020). RESTest: Black-Box Constraint-Based Testing of RESTful Web APIs. In: Kafeza, E., Benatallah, B., Martinelli, F., Hacid, H., Bouguettaya, A., Motahari, H. (eds) Service-Oriented Computing. ICSOC 2020. Lecture Notes in Computer Science(), vol 12571. Springer, Cham. https://doi.org/10.1007/978-3-030-65310-1_33
Download citation
DOI: https://doi.org/10.1007/978-3-030-65310-1_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-65309-5
Online ISBN: 978-3-030-65310-1
eBook Packages: Computer ScienceComputer Science (R0)