Interoperability and Integration Testing Methods for IoT Systems: A Systematic Mapping Study | SpringerLink
Skip to main content
Springer Nature Link
Log in

Interoperability and Integration Testing Methods for IoT Systems: A Systematic Mapping Study

  • Conference paper
  • First Online:
Software Engineering and Formal Methods (SEFM 2020)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 12310))

Included in the following conference series:

Abstract

The recent active development of Internet of Things (IoT) solutions in various domains has led to an increased demand for security, safety, and reliability of these systems. Security and data privacy are currently the most frequently discussed topics; however, other reliability aspects also need to be focused on to maintain smooth and safe operation of IoT systems. Until now, there has been no systematic mapping study dedicated to the topic of interoperability and integration testing of IoT systems specifically; therefore, we present such an overview in this study. We analyze 803 papers from four major primary databases and perform detailed assessment and quality check to find 115 relevant papers. In addition, recently published testing techniques and approaches are analyzed and classified; the challenges and limitations in the field are also identified and discussed. Research trends related to publication time, active researchers, and publication media are presented in this study. The results suggest that studies mainly focus only on general testing methods, which can be applied to integration and interoperability testing of IoT systems; thus, there are research opportunities to develop additional testing methods focused specifically on IoT systems, so that they are more effective in the IoT context.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 6291
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 7864
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Ahmad, A., Bouquet, F., Fourneret, E., Le Gall, F., Legeard, B.: Model-based testing as a service for IoT platforms. In: Margaria, T., Steffen, B. (eds.) ISoLA 2016. LNCS, vol. 9953, pp. 727–742. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-47169-3_55

    Chapter  Google Scholar 

  2. Ahmad, A., Bouquet, F., Fourneret, E., Legeard, B.: Model-based testing for internet of things systems. In: Advances in Computers, vol. 108, pp. 1–58. Elsevier (2018)

    Google Scholar 

  3. Ahmed, B.S., Bures, M., Frajtak, K., Cerny, T.: Aspects of quality in internet of things (IoT) solutions: a systematic mapping study. IEEE Access 7, 13758–13780 (2019)

    Article  Google Scholar 

  4. Ahmed, B.S., Zamli, K.Z., Afzal, W., Bures, M.: Constrained interaction testing: a systematic literature study. IEEE Access 5, 25706–25730 (2017)

    Article  Google Scholar 

  5. Aichernig, B.K., et al.: Learning a behavior model of hybrid systems through combining model-based testing and machine learning. In: Gaston, C., Kosmatov, N., Le Gall, P. (eds.) ICTSS 2019. LNCS, vol. 11812, pp. 3–21. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-31280-0_1

    Chapter  Google Scholar 

  6. Al-Fuqaha, A., Guizani, M., Mohammadi, M., Aledhari, M., Ayyash, M.: Internet of things: a survey on enabling technologies, protocols, and applications. IEEE Commun. Surv. Tutor. 17(4), 2347–2376 (2015)

    Article  Google Scholar 

  7. Amalfitano, D., Amatucci, N., De Simone, V., Riccio, V., Rita, F.A.: Towards a thing-in-the-loop approach for the verification and validation of IoT systems. In: Proceedings of the 1st ACM Workshop on the Internet of Safe Things, SafeThings 2017, pp. 57–63. ACM, New York (2017). https://doi.org/10.1145/3137003.3137007

  8. Anand, S., et al.: An orchestrated survey of methodologies for automated software test case generation. J. Syst. Softw. 86(8), 1978–2001 (2013)

    Article  Google Scholar 

  9. Arrieta, A., Sagardui, G., Etxeberria, L.: A model-based testing methodology for the systematic validation of highly configurable cyber-physical systems, October 2014

    Google Scholar 

  10. Bures, M.: PCTgen: automated generation of test cases for application workflows. In: Rocha, A., Correia, A.M., Costanzo, S., Reis, L.P. (eds.) New Contributions in Information Systems and Technologies. AISC, vol. 353, pp. 789–794. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16486-1_78

    Chapter  Google Scholar 

  11. Bures, M., Ahmed, B.S.: Employment of multiple algorithms for optimal path-based test selection strategy. Inf. Softw. Technol. 114, 21–36 (2019)

    Article  Google Scholar 

  12. Bures, M., Cerny, T., Ahmed, B.S.: Internet of things: current challenges in the quality assurance and testing methods. In: Kim, K.J., Baek, N. (eds.) ICISA 2018. LNEE, vol. 514, pp. 625–634. Springer, Singapore (2019). https://doi.org/10.1007/978-981-13-1056-0_61

    Chapter  Google Scholar 

  13. Camilli, M., Bellettini, C., Capra, L.: Design-time to run-time verification of microservices based applications. In: Cerone, A., Roveri, M. (eds.) SEFM 2017. LNCS, vol. 10729, pp. 168–173. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-74781-1_12

    Chapter  Google Scholar 

  14. Cavalcante, E., et al.: On the interplay of internet of things and cloud computing: a systematic mapping study. Comput. Commun. 89, 17–33 (2016)

    Article  Google Scholar 

  15. Cortés, M., Saraiva, R., Souza, M., Mello, P., Soares, P.: Adoption of software testing in internet of things: a systematic literature mapping. In: Proceedings of the IV Brazilian Symposium on Systematic and Automated Software Testing, pp. 3–11 (2019)

    Google Scholar 

  16. Cortés, M., Saraiva, R., Souza, M., Mello, P., Soares, P.: Adoption of software testing in internet of things: a systematic literature mapping. In: Proceedings of the IV Brazilian Symposium on Systematic and Automated Software Testing, SAST 2019, pp. 3–11. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3356317.3356326

  17. Datta, S.K., Bonnet, C., Baqa, H., Zhao, M., Le-Gall, F.: Approach for semantic interoperability testing in internet of things. In: 2018 Global Internet of Things Summit (GIoTS). IEEE, June 2018. https://doi.org/10.1109/giots.2018.8534582

  18. Dias, J.P., Couto, F., Paiva, A.C., Ferreira, H.S.: A brief overview of existing tools for testing the internet-of-things. In: 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 104–109. IEEE (2018)

    Google Scholar 

  19. Eckhart, M., Meixner, K., Winkler, D., Ekelhart, A.: Securing the testing process for industrial automation software. Comput. Secur. 85, 156–180 (2019). https://doi.org/10.1016/j.cose.2019.04.016

    Article  Google Scholar 

  20. Estivill-Castro, V., Hexel, R., Stover, J.: Modeling, validation, and continuous integration of software behaviours for embedded systems. In: 2015 IEEE European Modelling Symposium (EMS), pp. 89–95, October 2015. https://doi.org/10.1109/EMS.2015.24

  21. Felderer, M., Russo, B., Auer, F.: On testing data-intensive software systems. In: Biffl, S., Eckhart, M., Lüder, A., Weippl, E. (eds.) Security and Quality in Cyber-Physical Systems Engineering, pp. 129–148. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-25312-7_6

    Chapter  Google Scholar 

  22. Fernandes, J., et al.: IoT lab: towards co-design and IoT solution testing using the crowd. In: 2015 International Conference on Recent Advances in Internet of Things (RIoT), pp. 1–6, April 2015. https://doi.org/10.1109/RIOT.2015.7104907

  23. Garousi, V., Felderer, M., Karapıçak, Ç.M., Yılmaz, U.: Testing embedded software: a survey of the literature. Inf. Softw. Technol. 104, 14–45 (2018). https://doi.org/10.1016/j.infsof.2018.06.016

  24. Giraldo, J., Sarkar, E., Cardenas, A.A., Maniatakos, M., Kantarcioglu, M.: Security and privacy in cyber-physical systems: a survey of surveys. IEEE Des. Test 34(4), 7–17 (2017)

    Article  Google Scholar 

  25. Grace, P., Pickering, B., Surridge, M.: Model-driven interoperability: engineering heterogeneous IoT systems. Ann. Telecommun. 71(3–4), 141–150 (2015). https://doi.org/10.1007/s12243-015-0487-2

  26. GuŞeilă, L.G., Bratu, D., Moraru, S.: Continuous testing in the development of IoT applications. In: 2019 International Conference on Sensing and Instrumentation in IoT Era (ISSI), pp. 1–6 (2019)

    Google Scholar 

  27. Häser, F., Felderer, M., Breu, R.: Software paradigms, assessment types and non-functional requirements in model-based integration testing: a systematic literature review. In: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, p. 29. ACM (2014)

    Google Scholar 

  28. Incki, K., Ari, I.: Democratization of runtime verification for internet of things. Comput. Electr. Eng. 68, 570–580 (2018). https://doi.org/10.1016/j.compeleceng.2018.05.007

  29. Incki, K., Ari, I.: Observing interoperability of IoT systems through model-based testing. In: Fortino, G., et al. (eds.) InterIoT/SaSeIoT -2017. LNICST, vol. 242, pp. 60–66. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-93797-7_8

    Chapter  Google Scholar 

  30. Kaiser, A., Hackel, S.: Standards-based IoT testing with open-source test equipment. In: 2019 IEEE 19th International Conference on Software Quality, Reliability and Security Companion (QRS-C), pp. 435–441 (2019)

    Google Scholar 

  31. Khan, M.A., Salah, K.: IoT security: review, blockchain solutions, and open challenges. Future Gener. Comput. Syst. 82, 395–411 (2018)

    Article  Google Scholar 

  32. Kim, H., et al.: IoT-TaaS: towards a prospective IoT testing framework. IEEE Access 6, 15480–15493 (2018). https://doi.org/10.1109/access.2018.2802489

  33. Kiruthika, J., Khaddaj, S.: Software quality issues and challenges of internet of things. In: 2015 14th International Symposium on Distributed Computing and Applications for Business Engineering and Science (DCABES), pp. 176–179. IEEE (2015)

    Google Scholar 

  34. Kitchenham, B., Charters, S.: Guidelines for performing systematic literature reviews in software engineering (2007)

    Google Scholar 

  35. Larsen, K.G., Legay, A., Mikučionis, M., Nielsen, B., Nyman, U.: Compositional testing of real-time systems. In: Katoen, J.-P., Langerak, R., Rensink, A. (eds.) ModelEd, TestEd, TrustEd. LNCS, vol. 10500, pp. 107–124. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68270-9_6

    Chapter  Google Scholar 

  36. Lima, B.: Automated scenario-based integration testing of time-constrained distributed systems. In: 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST). IEEE, April 2019. https://doi.org/10.1109/icst.2019.00060

  37. Lu, A., Fang, W., Xu, C., Cheung, S.C., Liu, Y.: Data-driven testing methodology for RFID systems. Front. Comput. Sci. China 4(3), 354–364 (2010). https://doi.org/10.1007/s11704-010-0387-6

  38. Makedonski, P., et al.: Test descriptions with ETSI TDL. Softw. Qual. J. 27(2), 885–917 (2018). https://doi.org/10.1007/s11219-018-9423-9

  39. Medhat, N., Moussa, S., Badr, N., Tolba, M.F.: Testing techniques in IoT-based systems. In: 2019 Ninth International Conference on Intelligent Computing and Information Systems (ICICIS), pp. 394–401 (2019)

    Google Scholar 

  40. Nie, C., Leung, H.: A survey of combinatorial testing. ACM Comput. Surv. (CSUR) 43(2), 11 (2011)

    Article  Google Scholar 

  41. Park, H., Kim, H., Joo, H., Song, J.: Recent advancements in the internet-of-things related standards: a oneM2M perspective. ICT Express 2(3), 126–129 (2016). https://doi.org/10.1016/j.icte.2016.08.009

  42. Pontes, P.M., Lima, B., Faria, J.A.P.: Izinto: a pattern-based IoT testing framework. In: Companion Proceedings for the ISSTA/ECOOP 2018 Workshops, ISSTA 2018, pp. 125–131. ACM, New York (2018). https://doi.org/10.1145/3236454.3236511

  43. Pontes, P.M., Lima, B., Faria, J.A.P.: Test patterns for IoT. In: Proceedings of the 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation, A-TEST 2018, pp. 63–66. ACM, New York (2018). https://doi.org/10.1145/3278186.3278196

  44. Popereshnyak, S., Suprun, O., Suprun, O., Wieckowski, T.: IoT application testing features based on the modelling network. In: 2018 XIV-th International Conference on Perspective Technologies and Methods in MEMS Design (MEMSTECH), pp. 127–131, April 2018. https://doi.org/10.1109/MEMSTECH.2018.8365717

  45. Jaffar-ur Rehman, M., Jabeen, F., Bertolino, A., Polini, A.: Testing software components for integration: a survey of issues and techniques. Softw. Test. Verif. Reliab. 17(2), 95–133 (2007)

    Article  Google Scholar 

  46. Rings, T., Poglitsch, P., Schulz, S., Serazio, L., Vassiliou-Gioles, T.: A generic interoperability testing framework and a systematic development process for automated interoperability testing. Int. J. Softw. Tools Technol. Transf. 16(3), 295–313 (2013). https://doi.org/10.1007/s10009-013-0281-2

  47. Sand, B.: IoT testing - the big challenge why, what and how. In: Mandler, B., et al. (eds.) IoT360 2015. LNICST, vol. 170, pp. 70–76. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-47075-7_9

    Chapter  Google Scholar 

  48. Schieferdecker, I., Kretzschmann, S., Rennoch, A., Wagner, M.: IoT-testware - an eclipse project. In: 2017 IEEE International Conference on Software Quality, Reliability and Security (QRS). IEEE, July 2017. https://doi.org/10.1109/qrs.2017.59

  49. Shashank, S.P., Chakka, P., Kumar, D.V.: A systematic literature survey of integration testing in component-based software engineering. In: 2010 International Conference on Computer and Communication Technology (ICCCT), pp. 562–568. IEEE (2010)

    Google Scholar 

  50. Sotiriadis, S., Lehmets, A., Petrakis, E.G.M., Bessis, N.: Testing cloud services using the TestCast tool. In: Latifi, S. (ed.) Information Technology - New Generations. AISC, vol. 558, pp. 819–824. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-54978-1_101

    Chapter  Google Scholar 

  51. de Souza, B.P., Motta, R.C., de O. Costa, D., Travassos, G.H.: An IoT-based scenario description inspection technique. In: Proceedings of the XVIII Brazilian Symposium on Software Quality, SBQS 2019, pp. 20–29. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3364641.3364644

  52. de Souza, B.P., Motta, R.C., Travassos, G.H.: The first version of SCENARIotCHECK: a checklist for IoT based scenarios. In: Proceedings of the XXXIII Brazilian Symposium on Software Engineering, SBES 2019, pp. 219–223. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3350768.3350796

  53. Su, T., et al.: A survey on data-flow testing. ACM Comput. Surv. (CSUR) 50(1), 1–35 (2017)

    Article  MathSciNet  Google Scholar 

  54. Tan, T.-B., Cheng, W.-K.: Software testing levels in internet of things (IoT) architecture. In: Chang, C.-Y., Lin, C.-C., Lin, H.-H. (eds.) ICS 2018. CCIS, vol. 1013, pp. 385–390. Springer, Singapore (2019). https://doi.org/10.1007/978-981-13-9190-3_40

    Chapter  Google Scholar 

  55. Tönjes, R., Reetz, E.S., Moessner, K., Barnaghi, P.M.: A test-driven approach for life cycle management of internet of things enabled services. In: 2012 Future Network Mobile Summit (FutureNetw), pp. 1–8, July 2012

    Google Scholar 

  56. Walker, M.A., Schmidt, D.C., Dubey, A.: Testing at scale of IOT blockchain applications (chap. 6). In: Kim, S., Deka, G.C., Zhang, P. (eds.) Role of Blockchain Technology in IoT Applications, Advances in Computers, vol. 115, pp. 155–179. Elsevier (2019). https://doi.org/10.1016/bs.adcom.2019.07.008

Download references

Acknowledgement

This research is conducted as a part of the project TACR TH02010296 Quality Assurance System for the Internet of Things Technology. The authors acknowledge the support of the OP VVV funded project CZ.02.1.01/0.0/0.0/16_019 /0000765 “Research Center for Informatics”.

Author information

Authors and Affiliations

  1. Department of Computer Science, FEE, Czech Technical University in Prague, Karlovo namesti 13, Prague, Czech Republic

    Miroslav Bures, Matej Klima, Vaclav Rechtberger & Bestoun S. Ahmed

  2. Department of Electronic and Electrical Engineering, University of Strathclyde, 99 George Street, Glasgow, Scotland, UK

    Xavier Bellekens, Christos Tachtatzis & Robert Atkinson

  3. Department of Mathematics and Computer Science, Karlstad University, Universitetsgatan 2, 65188, Karlstad, Sweden

    Bestoun S. Ahmed

Authors
  1. Miroslav Bures
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Matej Klima
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Vaclav Rechtberger
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xavier Bellekens
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Christos Tachtatzis
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Robert Atkinson
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Bestoun S. Ahmed
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Miroslav Bures .

Editor information

Editors and Affiliations

  1. Informatica, Centrum voor Wiskunde en Informatica (CWI), Amsterdam, The Netherlands

    Frank de Boer

  2. Department of Computer Science, Nazarbayev University, Astana, Kazakhstan

    Antonio Cerone

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bures, M. et al. (2020). Interoperability and Integration Testing Methods for IoT Systems: A Systematic Mapping Study. In: de Boer, F., Cerone, A. (eds) Software Engineering and Formal Methods. SEFM 2020. Lecture Notes in Computer Science(), vol 12310. Springer, Cham. https://doi.org/10.1007/978-3-030-58768-0_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58768-0_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58767-3

  • Online ISBN: 978-3-030-58768-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation