Empirical evaluation of an educational game on software measurement | Empirical Software Engineering Skip to main content
Springer Nature Link
Log in

Empirical evaluation of an educational game on software measurement

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Software measurement is considered important in improving the software process. However, teaching software measurement remains a challenging issue. Although, games and simulations are regarded powerful tools for learning, their learning effectiveness is not rigorously established. This paper describes the results of an explorative study to investigate the learning effectiveness of a game prototype on software measurement in order to make an initial judgment about its potential as an educational tool as well as to analyze its appropriateness, engagement and strengths & weaknesses as guidance for further evolution. Within the study, a series of experiments was conducted in parallel in three master courses in Brazil. Results of the study reveal that the participants consider the content and structure of the game appropriate, but no indication for a significant difference on learning effectiveness could be shown.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  • Abt CC (2002) Serious games. University Press of America, Lanham, MD

    Google Scholar 

  • Akili GK (2007) Games and Simulations: A new approach in education. In: Gibson D, Aldrich C, Prensky M (eds) Games and simulations in online learning: research and development frameworks. Information Science Publishing, Hershey/PA, pp. 1–20

  • Almstrum VL, Dale N, Berglund A, Granger M, Currie Little J, Miller DM et al (1996) Evaluation: turning technology from toy to tool. Proceedings of the 1st Conference on Integrating Technology into Computer Science Education (ITiCSE '96). Barcelona, Spain, pp 201–217

  • Anderson LW, Krathwohl DR (eds) (2001) A taxonomy for learning, teaching, and assessing: a revision of bloom's taxonomy of educational objectives. Longman, New York

  • Baker A, Oh Navarro E, van der Hoek A (2003) Problems and programmers: an educational software engineering card game. Proceedings of the 2003 International Conference on Software Engineering. Portland, Oregon, pp 614–619

  • Basili VR, Caldiera G, Rombach HD (1994) Goal/question/metric approach. In: Marciniak J (ed) Encyclopedia of software engineering, vol. 1. John Wiley & Sons, New York, pp. 528–532

  • Bastien JMC, Scapin D (1993) Ergonomic criteria for the evaluation of human-computer interfaces. Technical report no. 156. Institut National de Recherche en Informatique et en Automatique, France

  • Bruns B, Gajewski P (1999) Multimediales Lernen im Netz: Leitfaden für Entscheider und Planer. Springer, Berlin (in German)

    Google Scholar 

  • Buglione L (2007) Project-o-poly. Giocare per Apprendere. Il gioco come opportunità nelle Learning Organizations. Persone & Conoscenze, Jan/Feb 2007, No.26/27, ESTE, pp 43–47 (in Italian)

  • Choi J, Hannafin M (1995) Situated cognition and learning environments: roles, structures and implications for design. Educ Technol Res Dev 43(2):53–69 doi:10.1007/BF02300472

    Article  Google Scholar 

  • CMMI Product Team (2006) CMMI for development, version 1.2. Technical report CMU/SEI-2006-TR-008. Software Engineering Institute/Carnegie Mellon University, Pittsburgh, Pennsylvania

  • Collofello JS (2000) University/industry collaboration in developing a simulation based software project management training course. Proceedings of the 13th Conference on Software Engineering Education and Training, Austin, Texas, pp 161–168

  • Dantas A, Barros M, Werner C (2004) A simulation-based game for project management experiential learning. Proceedings of the 16th International Conference on Software Engineering & Knowledge Engineering (SEKE'2004), Banff, Canada, pp 19–24

  • Dekkers CA, McQuaid PA (2002) The dangers of using software metrics to (Mis)Manage. IEEE IT Professional, IEEE Computer Society, March/April 2002

  • Drappa A, Ludewig J (2000) Simulation in software engineering training. Proceedings of the 22nd International Conference on Software Engineering. Limerick, Ireland, pp 199–208

  • Ellington H, Addinall E, Percival F (1982) A handbook of game design. Kogan Page, London

    Google Scholar 

  • Greitzer FL, Kuchar OA, Huston K (2007) Cognitive science implications for enhancing training effectiveness in a serious gaming context. J Educ Resour Comput 7(3) Art. 2, ACM/New York

  • Hock GT, Hui GLS (2004) A study of the problems and challenges of applying software metrics in software development industry. Proceedings of the M2USIC-MMU International Symposium on Information and Communication Technologies, Putrajaya, Malaysia

  • ISO 9241-151 (2008) Ergonomics of human-system interaction—part 151: guidance on world wide web user interfaces. International Organization for Standardization, Geneva

    Google Scholar 

  • Jain A, Boehm B (2006) SimVBSE: developing a game for value-based software engineering. Proceedings of the 19th Conference on Software Engineering Education and Training. Turtle Bay, Hawaii, pp 103–111

  • Janzen DS, Turner CS, Saiedian H (2007) Empirical software engineering in industry short courses. Proceedings of the 20th Conference on Software Engineering Education & Training (CSEET), Dublin, Ireland, pp 89–96

  • Kafai YB (2001) The educational potential of electronic games: from games-to-teach to games-to-learn. Conference on playing by the rules: the cultural policy challenges of video games. Chicago, Illinois

  • Kasunic M (2006) The state of software measurement practice: results of 2006 survey. Technical report CMU/SEI-2006-TR-009, Carnegie Mellon University/Software Engineering Institute, Pittsburgh, Pennsylvania

  • Kirkpatrick DL, Kirkpatrick JD (2006) Evaluating training programs: the four levels, 3rd edn. Berrett-Koehler Publishers, San Francisco, 379 pp

  • Levin J, Fox J (2006) Elementary statistics in social research. Allyn & Bacon, Boston, 384 pp

  • Lino JI (2007) Proposal of an educational game for software measurement and analysis. Project Thesis, Undergraduate Course on Information System, Federal University of Santa Catarina, Brazil (in Brazilian Portuguese)

  • Lipsey M (1990) Design sensitivity. Sage, California

    Google Scholar 

  • Löper S, Zehle M (2003) Evaluation of software metrics in the design phase and their implication on CASE tools. Master Thesis, Blekinge Institute of Technology, Sweden

  • Malone TW (1982) Heuristics for designing enjoyable user interfaces: Lessons from computer games. Proceedings of the Conference on Human factors in computing systems, Gaithersburg, Maryland

  • McGarry J, Card D, Jones C, Layman B, Clark E, Dean J et al (2001) Practical software measurement: objective information for decision makers. Addison-Wesley Professional, Reading

    Google Scholar 

  • McNabb M, Hawkes M, Rouk U (1999) Critical issues in evaluating the effectiveness of technology conference summary. National Conference on Educational Technology, Washington, D.C.

    Google Scholar 

  • Michael D, Chen S (2006) Serious games: games that educate, train, and inform. Thomson Course Technology, Boston

    Google Scholar 

  • Molenda M, Pershing JA, Reigeluth CM (1996) Designing instructional systems. In: Craig RL (ed) The ASTD training and development handbook. 4th edn. McGraw-Hill, New York, pp 266–293

    Google Scholar 

  • Oh Navarro E, van der Hoek A (2007) Comprehensive evaluation of an educational software engineering simulation environment. Proceedings of the 20th Conference on Software Engineering Education and Training, Dublin, Ireland, pp 195–202

  • Ott LM (2005) Developing healthy skepticism not disbelief-problems in teaching software metrics. Proceedings of the 1st Workshop on Methods for Learning Metrics at the 11th IEEE Software Metrics Symposium. Como, Italy

  • Percival F, Ellington H, Race P (1993) Handbook of educational technology, 3rd edn. Kogan Page, London

    Google Scholar 

  • Pfahl D, Laitenberger O, Dorsch J, Ruhe G (2003) An externally replicated experiment for evaluating the learning effectiveness of using simulations in software project management education. Empirical software engineering, v. 8. Kluwer Academic, The Netherlands, pp 367–395

    Google Scholar 

  • Prensky M (2001) Digital game-based learning. McGraw-Hill, New York

    Google Scholar 

  • Sharp H, Hall P (2000) An interactive multimedia software house simulation for postgraduate software engineers. Proceedings of the 22nd International Conference on Software Engineering. Limerick, Ireland, pp 688–691

  • Takona JP (2002) Educational research: principles and practice. Writers Club Press, New York, 604 pp

  • Thomas R (1996) A practical experiment in teaching software engineering metrics. Proceedings of the Int. Conference on Software Engineering: Education & Practice. Otago, New Zealand, pp 226–232

  • Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2000) Experimentation in software engineering—an introduction. Kluwer Academic, Norwell

    MATH  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Juliana I. Lino for her work on the conception of the initial version of the game and Leonardo Steil and Djoni Silva for the implementation of the prototype. A special thanks also to all the students of the master courses, who participated in the experiments. We would also like to thank Emily Oh Navarro for sharing material and we are grateful to Sílvia M. Nassar for advice on the statistical analysis. We would also like to thank the anonymous reviewers of a previous version of this paper for their valuable comments and suggestions.

This work was supported by the CNPq (Conselho Nacional de Desenvolvimento Científico e Tecnológico), an entity of the Brazilian government focused on scientific and technological development. Further support was provided by the UNIVALI—Universidade do Vale do Itajaí/Brazil.

Author information

Authors and Affiliations

  1. Universidade do Vale do Itajaí (UNIVALI), Master Program on Applied Computer Science, São José, SC, Brazil

    Christiane Gresse von Wangenheim, Marcello Thiry & Djone Kochanski

  2. Universidade Federal de Santa Catarina (UFSC) Graduate Program in Computer Science, Florianópolis, SC, Brazil

    Christiane Gresse von Wangenheim

Authors
  1. Christiane Gresse von Wangenheim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marcello Thiry
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Djone Kochanski
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christiane Gresse von Wangenheim.

Additional information

Editor: Forrest Shull

Appendix

Appendix

Table 11 Data on Y.1 Measurement competency
Full size table
Table 12 Data on Y.2 Measurement knowledge on the remembering level
Full size table
Table 13 Data on Y.3 Measurement knowledge on the understanding level
Full size table
Table 14 Data on Y.4 Measurement knowledge on the application level
Full size table
Table 15 Data on Y.5 Subjective learning effect
Full size table
Table 16 Data on Y.6 Appropriateness
Full size table
Table 17 Data on Y.7 Engagement
Full size table
Table 18 Data on DF1. Personal background
Full size table
Table 19 Data on DF.2 Motivation
Full size table
Table 20 Data on DF3. Additional study time
Full size table

Rights and permissions

Reprints and permissions

About this article

Cite this article

Gresse von Wangenheim, C., Thiry, M. & Kochanski, D. Empirical evaluation of an educational game on software measurement. Empir Software Eng 14, 418–452 (2009). https://doi.org/10.1007/s10664-008-9092-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-008-9092-6

Keywords

Search

Navigation