Impact of Gamification on Trace Link Vetting: A Controlled Experiment | SpringerLink
Skip to main content
Springer Nature Link
Log in

Impact of Gamification on Trace Link Vetting: A Controlled Experiment

  • Conference paper
  • First Online:
Requirements Engineering: Foundation for Software Quality (REFSQ 2019)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 11412))

  • 1703 Accesses

Abstract

[Context] Automatically generated trace links must be vetted by human analysts before use. The task of vetting trace links is considered boring due to its repetitive nature and tools that are not engaging to the analyst. Therefore, a lack of developer engagement can hamper the successful implementation of a traceability strategy in an organisation. [Objective] In this study, we examine whether two gamification features, levels and badges, have a positive effect on human analysts’ engagement and ultimately on the quality of vetted trace links. [Method] We have conducted a controlled experiment with 24 participants that vetted trace link candidates and recorded their speed, correctness, enjoyment, and perceived usability of the tool. [Results] The results indicate that there was no significant difference between the speed, correctness, and perceived usability of the control and the experiment group. However, gamification features significantly increased the users’ perceived enjoyment. Levels and badges were perceived positively by the majority of the participants while some pitfalls and improvements were pointed out. [Conclusion] Our study indicates the need for further research as the results raise several questions, in particular w.r.t. what analyst behaviour gamification incentivises, and the impact of gamification on long-term enjoyment.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 7549
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 9437
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Bangor, A., Kortum, P., Miller, J.: Determining what individual SUS scores mean: adding an adjective rating scale. J. Usability Stud. 4(3), 114–123 (2009)

    Google Scholar 

  2. Borg, M., Runeson, P., Ardö, A.: Recovering from a decade: a systematic mapping of information retrieval approaches to software traceability. Empir. Softw. Eng. (ESE) 19(6), 1565–1616 (2014)

    Article  Google Scholar 

  3. Brooke, J., et al.: SUS-a quick and dirty usability scale. Usability Eval. Ind. 189(194), 4–7 (1996)

    Google Scholar 

  4. Charles, D., Charles, T., McNeill, M., Bustard, D., Black, M.: Game-based feedback for educational multi-user virtual environments. Br. J. Educ. Technol. 42(4), 638–654 (2011)

    Article  Google Scholar 

  5. Cleland-Huang, J., Czauderna, A., Gibiec, M., Emenecker, J.: A machine learning approach for tracing regulatory codes to product specific requirements. In: 32nd ACM/IEEE International Conference on Software Engineering ICSE 2010, pp. 155–164 (2010)

    Google Scholar 

  6. Cuddeback, D., Dekhtyar, A., Hayes, J.: Automated requirements traceability: the study of human analysts. In: RE 2010, pp. 231–240. IEEE (2010)

    Google Scholar 

  7. Cuddeback, D., Dekhtyar, A., Hayes, J.H., Holden, J., Kong, W.K.: Towards overcoming human analyst fallibility in the requirements tracing process. In: ICSE 2011, pp. 860–863. ACM (2011)

    Google Scholar 

  8. Dalpiaz, F., Snijders, R., Brinkkemper, S., Hosseini, M., Shahri, A., Ali, R.: Engaging the crowd of stakeholders in requirements engineering via gamification. In: Stieglitz, S., Lattemann, C., Robra-Bissantz, S., Zarnekow, R., Brockmann, T. (eds.) Gamification. PI, pp. 123–135. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-45557-0_9

    Chapter  Google Scholar 

  9. Deci, E.L., Koestner, R., Ryan, R.M.: A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychol. Bull. 125(6), 627 (1999)

    Article  Google Scholar 

  10. Dekhtyar, A., Dekhtyar, O., Holden, J., Hayes, J.H., Cuddeback, D., Kong, W.K.: On human analyst performance in assisted requirements tracing: statistical analysis. In: RE 2011, pp. 111–120. IEEE (2011)

    Google Scholar 

  11. Dubois, D.J., Tamburrelli, G.: Understanding gamification mechanisms for software development. In: FSE 2013, pp. 659–662. ACM (2013)

    Google Scholar 

  12. Eickhoff, C., Harris, C.G., de Vries, A.P., Srinivasan, P.: Quality through flow and immersion: gamifying crowdsourced relevance assessments. In: Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 871–880. ACM (2012)

    Google Scholar 

  13. Falessi, D., Juristo, N., Wohlin, C., Turhan, B., Münch, J., Jedlitschka, A., Oivo, M.: Empirical software engineering experts on the use of students and professionals in experiments. Empir. Softw. Eng. 23(1), 452–489 (2018)

    Article  Google Scholar 

  14. García, F., Pedreira, O., Piattini, M., Cerdeira-Pena, A., Penabad, M.: A framework for gamification in software engineering. J. Syst. Softw. 132, 21–40 (2017)

    Article  Google Scholar 

  15. Hamari, J., Koivisto, J., Sarsa, H.: Does gamification work? – A literature review of empirical studies on gamification. In: 47th Hawaii International Conference on System Sciences (HICSS), pp. 3025–3034. IEEE (2014)

    Google Scholar 

  16. Hanus, M.D., Fox, J.: Assessing the effects of gamification in the classroom: a longitudinal study on intrinsic motivation, social comparison, satisfaction, effort, and academic performance. Comput. Educ. 80, 152–161 (2015)

    Article  Google Scholar 

  17. Hayes, J.H., Dekhtyar, A., Larsen, J., Guéhéneuc, Y.G.: Effective use of analysts’ effort in automated tracing. Requir. Eng. 23(1), 119–143 (2018)

    Article  Google Scholar 

  18. Hayes, J.H., Dekhtyar, A., Osborne, J.: Improving requirements tracing via information retrieval. In: RE 2003, pp. 138–147. IEEE (2003)

    Google Scholar 

  19. Hayes, J.H., Dekhtyar, A., Sundaram, S.: Text mining for software engineering: how analyst feedback impacts final results. In: ACM SIGSOFT Software Engineering Notes, vol. 30, pp. 1–5. ACM (2005)

    Google Scholar 

  20. Kappen, D.L., Nacke, L.E.: The kaleidoscope of effective gamification: deconstructing gamification in business applications. In: Proceedings of the 1st International Conference on Gameful Design, Research, and Applications, pp. 119–122. ACM (2013)

    Google Scholar 

  21. Kifetew, F.M., et al.: Gamifying collaborative prioritization: does pointsification work? In: RE 2017, pp. 322–331. IEEE (2017)

    Google Scholar 

  22. Kolpondinos, M.Z.H., Glinz, M.: Behind points and levels–the influence of gamification algorithms on requirements prioritization. In: RE 2017, pp. 332–341. IEEE (2017)

    Google Scholar 

  23. Kong, W.K., Hayes, J.H., Dekhtyar, A., Dekhtyar, O.: Process improvement for traceability: a study of human fallibility. In: RE 2012, pp. 31–40. IEEE (2012)

    Google Scholar 

  24. Kong, W.K., Huffman Hayes, J., Dekhtyar, A., Holden, J.: How do we trace requirements: an initial study of analyst behavior in trace validation tasks. In: Proceedings of the 4th International Workshop on Cooperative and Human Aspects of Software Engineering, pp. 32–39. ACM (2011)

    Google Scholar 

  25. Lombriser, P., Dalpiaz, F., Lucassen, G., Brinkkemper, S.: Gamified requirements engineering: model and experimentation. In: Daneva, M., Pastor, O. (eds.) REFSQ 2016. LNCS, vol. 9619, pp. 171–187. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-30282-9_12

    Chapter  Google Scholar 

  26. Maro, S., Steghöfer, J.P.: Capra: a configurable and extendable traceability management tool. In: RE 2016, pp. 407–408. IEEE (2016)

    Google Scholar 

  27. Maro, S., Steghöfer, J.P., Huffman Hayes, J., Cleland-Huang, J., Staron, M.: Vetting automatically generated trace links: what information is useful to human analysts? In: RE 2018, pp. 52–63. IEEE (2018)

    Google Scholar 

  28. Maro, S., Sundklev, E., Persson, C.O., Liebel, G., Steghöfer, J.P.: Impact of gamification on trace link vetting: a controlled experiment, January 2019. https://doi.org/10.5281/zenodo.2540646. Dataset

  29. Mekler, E.D., Brühlmann, F., Opwis, K., Tuch, A.N.: Do points, levels and leaderboards harm intrinsic motivation?: an empirical analysis of common gamification elements. In: Proceedings of the 1st International Conference on Gameful Design, Research, and Applications. pp. 66–73. ACM (2013)

    Google Scholar 

  30. Morschheuser, B., Hamari, J., Werder, K., Abe, J.: How to gamify? A method for designing gamification (2017)

    Google Scholar 

  31. Parizi, R.M.: On the gamification of human-centric traceability tasks in software testing and coding. In: Software Engineering Research, Management and Applications (SERA), pp. 193–200. IEEE (2016)

    Google Scholar 

  32. Pedreira, O., García, F., Brisaboa, N., Piattini, M.: Gamification in software engineering-a systematic mapping. Inf. Softw. Technol. 57, 157–168 (2015)

    Article  Google Scholar 

  33. Schütze, H., Manning, C.D., Raghavan, P.: Introduction to Information Retrieval, vol. 39. Cambridge University Press, Cambridge (2008)

    MATH  Google Scholar 

  34. Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A.: Experimentation in Software Engineering. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-29044-2

    Book  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Software Engineering Division, Chalmers | University of Gothenburg, Gothenburg, Sweden

    Salome Maro, Emil Sundklev, Carl-Oscar Persson & Jan-Philipp Steghöfer

  2. School of Computer Science, Reykjavik University, Reykjavik, Iceland

    Grischa Liebel

Authors
  1. Salome Maro
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Emil Sundklev
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Carl-Oscar Persson
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Grischa Liebel
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jan-Philipp Steghöfer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Salome Maro or Grischa Liebel .

Editor information

Editors and Affiliations

  1. Department of Computer Science and Engineering, Chalmers University of Technology, Gothenburg, Sweden

    Eric Knauss

  2. paluno – The Ruhr Institute for Software Technology, University of Duisburg-Essen, Essen, Germany

    Michael Goedicke

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Maro, S., Sundklev, E., Persson, CO., Liebel, G., Steghöfer, JP. (2019). Impact of Gamification on Trace Link Vetting: A Controlled Experiment. In: Knauss, E., Goedicke, M. (eds) Requirements Engineering: Foundation for Software Quality. REFSQ 2019. Lecture Notes in Computer Science(), vol 11412. Springer, Cham. https://doi.org/10.1007/978-3-030-15538-4_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-15538-4_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-15537-7

  • Online ISBN: 978-3-030-15538-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics