Comparing Graph-Based Algorithms to Generate Test Cases from Finite State Machines | Journal of Electronic Testing Skip to main content

Advertisement

Springer Nature Link
Log in

Comparing Graph-Based Algorithms to Generate Test Cases from Finite State Machines

  • Published:
Journal of Electronic Testing Aims and scope Submit manuscript

Abstract

Model-Based Testing (MBT) is a well-known technique that employs formal models to represent reactive systems’ behavior and generates test cases. Such systems have been specified and verified using mostly Finite State Machines (FSMs). There is a plethora of test generation algorithms in the literature; most of them are based on graphs once an FSM can be formally defined as a graph. Nevertheless, there is a lack of studies on analyzing cost and efficiency of FSM-based test generation algorithms. This study compares graph-based algorithms adopted to generate test cases from FSM models. In particular, we compare the Chinese Postman Problem (CPP) and H-Switch Cover (HSC) algorithms with the well-known Depth-First Search (DFS) and Breadth-First Search (BFS) algorithms in the context of covering all-transitions and all-transition-pairs criteria in an FSM. First, a systematic literature mapping was conducted to summarize the methods that have been adopted in MBT, considering FSMs. Second, the main methods found were implemented and analyzed on randomly-generated FSMs, as well as real-world models that represent embedded systems of space applications. To make comparisons, we considered analyses in terms of cost (time), efficiency (mutant analysis) and characteristics of the generated test suites (number of test cases, average length of test cases, largest and smallest test cases, standard deviation and distribution of test cases). In general, CPP presented the best results in terms of number of test cases and test suite size. In addition, CPP also presented low distribution of average length compared to other algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23

Similar content being viewed by others

Notes

  1. Mutation analysis is a technique that artificially introduces faults in a program, generating modified versions called mutants. Such set of mutants can be used as to build an indicator of fault detection capabilities of a test suite.

  2. https://www.scopus.com/home.uri

  3. The references of the 97 selected studies can be accessed at: https://goo.gl/Yv1KKn.

  4. https://github.com/MatheusMMariano/H-Switch_Cover

  5. http://www.inpe.br/

  6. https://webperformcharts.lcc.unifesspa.edu.br/nova.html

References

  1. Ali S, Briand LC, Rehman MJ-U, Asghar H, Iqbal MZZ, Nadeem A (2007) A state-based approach to integration testing based on uml models. Inf Softw Technol 49(11-12):1087–1106

    Article  Google Scholar 

  2. Amaral A (2005) Geração de casos de testes para sistemas especificados em statecharts. (INPE-14215-TDI/1116)

  3. Ammann P, Offutt J (2008) Introduction to Software Testing. Cambridge University Press, Cambridge. [S.l.]

    Book  Google Scholar 

  4. Anbunathan R, Basu A (2013) Dataflow test case generation from uml class diagrams. In: Proc. IEEE international conference on computational intelligence and computing research, 2013. IEEE, pp 1–?9

  5. Arantes AO, de Santiago VA, Vijaykumar NL, De Souza EF (2014) Tool support for generating model-based test cases via web. Int J Web Eng Technol iiWAS 9(1):62–96

    Article  Google Scholar 

  6. Barmi ZA, Ebrahimi AH, Feldt R (2011) Alignment of requirements specification and testing: a systematic mapping study. In: Proc. international conference on software testing, verification and validation workshops, 2011. IEEE, pp 476?-485

  7. Belli F, Beyazit M, Takagi T, Furukawa Z (2012) Model-based mutation testing using pushdown automata. IEICE Trans Inf Syst 95(9):2211–2218

    Article  Google Scholar 

  8. Belli F, Beyazit M, Takagi T, Furukawa Z (2013) Mutation testing of “go-back” functions based on pushdown automata. In: Proc. international conference on software testing, verification and validation, 2011. IEEE, pp 249–258

  9. Belli F, Endo AT, Linschulte M, Simao A (2014) A holistic approach to model-based testing of web service compositions. Software: Practice and Experience 44(2):201–234

    Google Scholar 

  10. Belli F, Hollmann A, Kleinselbeck M (2009) A graph-model-based testing method compared with the classification tree method for test case generation. In: Proc. international conference on secure software integration and reliability improvement, 3., 2009. IEEE, pp 193–200

  11. Bondy J, Murty U (2008) Graduate texts in mathematics: graph theory. Springer, USA

    Google Scholar 

  12. Brito RC, Martendal DM, de Oliveira HEM (2003) Máquinas de estados finitos de mealy e moore

  13. Broy M, Jonsson B, Katoen JP, Leucker M, Pretschner A (2005) Model-based testing of reactive systems. Springer, Berlin

    Book  MATH  Google Scholar 

  14. Cartaxo EG, Machado PD, Neto FGO (2011) On the use of a similarity function for test case selection in the context of model-based testing. Software Testing, Verification and Reliability 21(2):75–100

    Article  Google Scholar 

  15. Chow TS (1978) Testing software design modeled by finite-state machines. IEEE Trans Softw Eng SE-4 (3):178–187

    Article  MATH  Google Scholar 

  16. Dang T, Nahhal T (2009) Coverage-guided test generation for continuous and hybrid systems. Formal Methods in System Design 34(2):183–213

    Article  MATH  Google Scholar 

  17. Devroey X, Perrouin G, Schobbens P-Y (2014) Abstract test case generation for behavioural testing of software product lines, 86–93

  18. Dorofeeva R, El-Fakih K, Maag S, Cavalli AR, Yevtushenko N (2010) Fsm-based conformance testing methods: a survey annotated with experimental evaluation. Inf Softw Technol 52(12):1286–1297

    Article  Google Scholar 

  19. Dorofeeva R, El-Fakih K, Yevtushenko N (2005) An improved conformance testing method. In: Proc. international conference on formal techniques for networked and distributed systems, 2005, pp 204–218

  20. Edmonds J, Johnson EL (1973) Matching, euler tours and the chinese postman. Math Program 5(1):88–124

    Article  MathSciNet  MATH  Google Scholar 

  21. Endo AT, Simao A (2013) Evaluating test suite characteristics, cost, and effectiveness of fsm-based testing methods. Inf Softw Technol 55(6):1045–1062

    Article  Google Scholar 

  22. Felizardo KR, Mendes E, Kalinowski M, Souza EF, Vijaykumar NL (2016) Using forward snowballing to update systematic reviews in software engineering. In: International symposium on empirical software engineering and measurement (ESEM)

  23. Gill A, et al. (1962) Introduction to the theory of finite-state machines. [S.I.] McGraw-Hill, New York

    MATH  Google Scholar 

  24. Gonenc G (1970) A method for the design of fault detection experiments. IEEE Trans Comput 100(6):551–558

    Article  MATH  Google Scholar 

  25. Gurbuz HG, Tekinerdogan B (2017) Model-based testing for software safety: a systematic mapping study. Softw Qual J 26(4):1–46

    Google Scholar 

  26. Hessel A, Pettersson P (2007) A global algorithm for model-based test suite generation. Electronic Notes in Theoretical Computer Science 190(2):47–59

    Article  Google Scholar 

  27. IEEE (2005) Ieee standard for software verification and validation. IEEE Std 1012-2004 (Revision of IEEE Std 1012-1998), pp 1–110

  28. Jia Y, Harman M (2011) An analysis and survey of the development of mutation testing. Trans Softw Eng 37(5):649–678

    Article  Google Scholar 

  29. Just R, Jalali D, Inozemtseva L, Ernst MD, Holmes R, Fraser G (2014) Are mutants a valid substitute for real faults in software testing?. In: Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering, (FSE-22), Hong Kong, China, November 16 - 22, 2014. ACM, pp 654– 665

  30. Khan MU, Iftikhar S, Iqbal MZ, Sherin S (2017) Empirical studies omit reporting necessary details: a systematic literature review of reporting quality in model based testing. Computer Standards & Interfaces

  31. Kitchenham BA (2007) Guidelines for performing systematic literature reviews in software engineering. Keele University, Durham. Technical Report EBSE-2007-01

  32. Lee D, Yannakakis M (1996) Principles and methods of testing finite state machines-a survey. Proc IEEE 84(8):1090–1123

    Article  Google Scholar 

  33. Li N, Offutt J (2017) Test oracle strategies for model-based testing. IEEE Trans Softw Eng 43(4):372–395

    Article  Google Scholar 

  34. Majeed S, Ryu M (2016) Model-based replay testing for event-driven software. In: Proceedings..., pages 1527–1533. Annual ACM Symposium on Applied Computing, 31., 2016, ACM

  35. Mariano MM, Souza ÉF, Endo AT, Vijaykumar NL (2016) A comparative study of algorithms for generating switch cover test sets. In: Anais..., Maceió. Simpósio Brasileiro de Qualidade de Software, 15., 2016

  36. Mariano MM, Souza EF, Endo AT, Vijaykumar NL (2019a) Analyzing graph-based algorithms employed to generate test cases from finite state machines. In: 2019 IEEE Latin American test symposium (LATS), pp 1–6

  37. Mariano MM, Souza EF, Endo AT, Vijaykumar NL (2019) Identifying approaches to generate test cases in model-based testing: a systematic mapping.. In: Proceedings of the Iberoamerican conference on software engineering, 22., 2019, pp 1–14, Havana

  38. Mathur AP (2008) Foundations of software testing. [S.l.] Pearson Education, London

    Google Scholar 

  39. Myers GJ, Sandler C, Badgett T (1976) The art of software testing. [S.l.] Wiley, New York

    Google Scholar 

  40. Naito S (1981) Fault detection for sequential machines by transition-tours

  41. Neumann F (2004) Expected runtimes of evolutionary algorithms for the eulerian cycle problem. In: Proceedings..., pages 904–910. Congress on Evolutionary Computation, 2001. IEEE

  42. Papadakis M, Shin D, Yoo S, Bae D (2018) Are mutation scores correlated with real fault detection?: a large scale empirical study on the relationship between mutants and real faults. In: Proceedings of the 40th international conference on software engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018, pp 537–548. ACM

  43. Petersen K, Vakkalanka S, Kuzniarz L (2015) Guidelines for conducting systematic mapping studies in software engineering: an update. IEEE Trans Vis Comput Graph 64:1–18

    Google Scholar 

  44. Pimont S, Rault J-C (1976) A software reliability assessment based on a structural and behavioral analysis of programs. In: Proc. international conference on software engineering, 2., 1976, IEEE, pp 486–491

  45. Pinheiro AC, Simão A, Ambrosio AM (2014) Fsm-based test case generation methods applied to test the communication software on board the itasat university satellite: a case study. J Aerosp Technol Manag 6(4):447–461

    Article  Google Scholar 

  46. Pressman RS (2005) Software engineering: a practitioner’s approach. [S.l.]: Palgrave Macmillan

  47. Proch S, Mishra P (2014) Directed test generation for hybrid systems. In: Proceedings..., pages 156–162. International Symposium on Quality Electronic Design, 15., 2014

  48. Santiago V, Vijaykumar NL, Guimarães D, Amaral AS, Ferreira É (2008) An environment for automated test case generation from statechart-based and finite state machine-based behavioral models. In: Proc. Software Testing Verification and Validation Workshop, 2008. ICSTW’08. IEEE International Conference on, pp 63–72

  49. Satpathy M, Yeolekar A, Peranandam P, Ramesh S (2012) Efficient coverage of parallel and hierarchical stateflow models for test case generation. Software Testing, Verification and Reliability 22(7):457–479

    Article  Google Scholar 

  50. Siavashi F, Truscan D (2014) Environment modeling in model-based testing: concepts, prospects and research challenges: a systematic literature review. In: Proceedings..., pages 30:1–30:6. Proc. International Conference on Evaluation and Assessment in Software Engineering, 19., 2015. ACM, New York

  51. Sidhu DP, Leung T-K (1989) Formal methods for protocol testing: a detailed study. IEEE Trans Softw Eng 15(4):413–426

    Article  Google Scholar 

  52. Simão A., Petrenko A, Maldonado J (2009) Comparing finite state machine test coverage criteria. 3(2): 91–105

  53. Souza ÉF, Santiago VA JR, Guimaraes D, Vijaykumar NL (2008) Evaluation of test criteria for space application software modeling in statecharts. In: Proc. international conference on computational intelligence for modelling control and automation, p 2008

  54. Souza ÉF, Santiago VA JR, Vijaykumar NL (2017) H-switch cover: a new test criterion to generate test case from finite state machines. Softw Qual J 25(2):373–405

    Article  Google Scholar 

  55. Souza SDRSD (2000) Validação de especificações de sistemas reativos: definição e análise de critérios de teste

  56. Vu T-D, Hung PN, Nguyen V-H (2015) A method for automated test data generation from sequence diagrams and object constraint language, pp 335–341

  57. Wang W, Sampath S, Lei Y, Kacker R, Kuhn R, Lawrence J (2016) Using combinatorial testing to build navigation graphs for dynamic web applications. Software testing. Verification and Reliability 26(4):318–346

    Article  Google Scholar 

  58. Yao J, Wang Z, Yin X, Shiyz X, Wu J (2014) Formal modeling and systematic black-box testing of sdn data plane. In: Proc. international conference on network protocols, 22., 2014, IEEE, pp 179–190

Download references

Acknowledgments

The authors would like to thank CAPES for the financial support and Dr. Rafael Santos for his help with the analyses. A.T. Endo and E.F. Souza are partially financially supported by CNPq/Brazil (grant numbers 420363/2018-1 and 432247/2018-1, respectively).

Author information

Authors and Affiliations

  1. Laboratory of Computing and Applied Mathematics, National Institute for Space Research, INPE Av. dos Astronautas, 1.758 - Jardim da Granja, São José dos Campos, SP, 12227-010, Brazil

    Matheus Monteiro Mariano

  2. Department of Computer, Federal University of Technology, UTFPR Av. Alberto Carazzai, 1640, Cornélio Procópio, PR, 86300-000, Brazil

    Érica Ferreira de Souza & André Takeshi Endo

  3. ICT - Institute of Science & Technology, Federal University of São Paulo, Av. Cesare Monsueto Giulio Lattes, 1201, São José dos Campos, Brazil

    Nandamudi Lankalapalli Vijaykumar

  4. LABAC, National Institute for Space Research, Av. dos Astronautas, 1758, São José dos Campos, Brazil

    Nandamudi Lankalapalli Vijaykumar

Authors
  1. Matheus Monteiro Mariano
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Érica Ferreira de Souza
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. André Takeshi Endo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nandamudi Lankalapalli Vijaykumar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Matheus Monteiro Mariano.

Additional information

Responsible Editor: L.M.B. Pö hls

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mariano, M.M., de Souza, É., Endo, A.T. et al. Comparing Graph-Based Algorithms to Generate Test Cases from Finite State Machines. J Electron Test 35, 867–885 (2019). https://doi.org/10.1007/s10836-019-05844-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10836-019-05844-6

Keywords