Abstract
Auxiliary functions in software systems, often overlooked due to their perceived simplicity, play a crucial role in overall system reliability. This study focuses on the effectiveness of agile practices, specifically the pair programming and the test-first programming practices. Despite the importance of these functions, there exists a dearth of empirical evidence on the impact of agile practices on their development, raising questions about their potential to enhance correctness without affecting time-to-market. This paper aims to bridge this gap by comparing the application of agile practices with traditional approaches in the context of auxiliary function development. We conducted six experiments involving 122 participants (85 novices and 37 professionals) who used both traditional and agile methods to develop six auxiliary functions across three different domains. Our analysis of 244 implementations suggests the potential benefits of agile practices in auxiliary function development. Pair programming showed a tendency towards improved correctness, while test-first programming did not significantly extend the total development time, particularly among professionals. However, these findings should be interpreted cautiously as they do not conclusively establish that agile practices outperform traditional approaches universally. As indicated by our results, the potential benefits of agile practices may vary depending on factors such as the programmer’s experience level and the nature of the functions being developed. Further research is needed to fully understand the contexts in which these practices can be most effectively applied and to address the potential limitations of our study.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Availability of data and materials
The authors declare that the data (programs, test sets, function descriptions, and more) supporting the findings of this study are available within the paper and are accessible via the following link: http://www.ict.unifesp.br/fsilveira/data/SoftwareTestingExperiment_data.zip.
Notes
http://junit.org/ – accessed in July, 2023.
Duration typically refers to the total time required for individuals, pairs, and teams of developers to complete tasks.
Effort is generally calculated by summing the time spent by each individual in a pair or team of developers, akin to our method of measuring effort, i.e., considering twice the time spent for pairs of programmers.
Only a single subject that participated in our study did not respond to our survey and was thus not considered in the graphs.
http://commons.apache.org/ – accessed in July, 2023.
This function is cited because its source code is available online. Nonetheless, it is reasonable to presume that the other functions listed in Sect. 1 are also of equivalent size. For example, a PlayStation-like leap year detection function requires only a small number of code lines to be implemented.
All experimental artifacts, encompassing subjects’ programs, test sets, function descriptions, and more, are accessible via the following link: http://www.ict.unifesp.br/fsilveira/data/SoftwareTestingExperiment_data.zip.
http://eclipse.org/ – accessed in July, 2023.
http://cobertura.github.io/cobertura/ – accessed in July, 2023.
http://www.r-project.org/ - 07/06/2021.
References
Abrahamsson, P., Hanhineva, A., & Jäälinoja, J. (2005). Improving business agility through technical solutions: a case study on test-driven development in mobile software development. In R. Baskerville, L. Mathiassen, J. Pries-Heje, & J. DeGross (Eds.), IFIP Advances in Information and Communication Technology (pp. 227–243). Germany: Springer.
Arisholm, E., Gallis, H., Dyba, T., & Sjoberg, D. I. (2007). Evaluating pair programming with respect to system complexity and programmer expertise. IEEE Transactions on Software Engineering, 33, 65–86. https://doi.org/10.1109/TSE.2007.17
Baldassarre, M. T., Caivano, D., Fucci, D., Juristo, N., Romano, S., Scanniello, G., & Turhan, B. (2021). Studying test-driven development and its retainment over a six-month time span. Journal of Systems and Software, 176, 110937. https://doi.org/10.1016/j.jss.2021.110937
Bavota, G., & Russo, B. (2015). Four eyes are better than two: On the impact of code reviews on software quality. 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME) (pp. 81–90). https://doi.org/10.1109/ICSM.2015.7332454
Beck, K. (2002). Test Driven Development: By Example. Inc, USA: Addison-Wesley Longman Publishing Co.
Begel, A., & Simon, B. (2008). Novice software developers, all over again. Proc. of the ICER ’08 (pp. 3–14). New York, NY, USA: ACM.
Bella, E., Fronza, I., Phaphoom, N., Sillitti, A., Succi, G., & Vlasenko, J. (2013). Pair programming and software defects-a large, industrial case study. IEEE Transactions on Software Engineering, 39(7), 930–953. https://doi.org/10.1109/TSE.2012.68
Bissi, W., Serra Seca Neto, A. G., & Emer, M. C. F. P. (2016). The effects of test driven development on internal quality, external quality and productivity: A systematic review. Information and Software Technology, 74, 45–54. https://doi.org/10.1016/j.infsof.2016.02.004
Black Duck Software Inc. (2014). Open Hub Code Search. https://www.openhub.net/. Accessed 07 Jun 2021
Bland, M. (2014) Goto Fail, Heartbleed, and Unit Testing Culture. http://martinfowler.com/articles/testing-culture.html. Accessed 07 Jun 2021
Brock, Z. (2015) What tech companies do a lot of pair programming? http://qr.ae/Rkh8Gl. Accessed 07 Jun 2021
Canfora, G., Cimitile, A., Garcia, F., Piattini, M., & Visaggio, C. A. (2007). Evaluating performances of pair designing in industry. Journal of Systems and Software, 80, 1317–1327. https://doi.org/10.1016/j.jss.2006.11.004
Cellan-Jones, R. (2010). Sony’s leap year bug. http://www.bbc.co.uk/blogs/thereporters/rorycellanjones/2010/03/sonys_millennium_bug.html. Accessed 07 Jun 2021
da Silva Estácio, B. J., & Prikladnicki, R. (2015). Distributed pair programming: A systematic literature review. Information and Software Technology, 63, 1–10. https://doi.org/10.1016/j.infsof.2015.02.011
Dagenais, B., Ossher, H., Bellamy, R. K. E., Robillard, M. P., & Vries, J. P. (2010). Moving into a new software project landscape. Proc. of the ICSE ’10 (pp. 275–284). New York, NY, USA: ACM.
Demir, Ö., & Seferoglu, S. S. (2021). A comparison of solo and pair programming in terms of flow experience, coding quality, and coding achievement. Journal of Educational Computing Research, 58(8), 1448–1466. https://doi.org/10.1177/0735633120949788
Desai, C., Janzen, D., & Savage, K. (2008). A survey of evidence for test-driven development in academia. ACM SIGCSE Bulletin, 40, 97–101. https://doi.org/10.1145/1383602.1383644
Dingsøyr, T., Nerur, S., Balijepally, V., & Moe, N. (2012). A decade of agile methodologies: Towards explaining agile software development. Journal of Systems and Software, 85, 1213–1221. https://doi.org/10.1016/j.jss.2012.02.033
Edwards, S. H. (2004). Using software testing to move students from trial-and-error to reflection-in-action. SIGCSE Bulletin, 36, 26–30. https://doi.org/10.1145/1028174.971312
Erdogmus, H., Morisio, M., & Torchiano, M. (2005). On the effectiveness of the test-first approach to programming. IEEE Transactions on Software Engineering, 31, 226–237. https://doi.org/10.1109/TSE.2005.37
Fucci, D., Erdogmus, H., Turhan, B., Oivo, M., & Juristo, N. (2017). A dissection of the test-driven development process: Does it really matter to test-first or to test-last? IEEE Transactions on Software Engineering, 43(7), 597–614. https://doi.org/10.1109/TSE.2016.2616877
Fucci, D., Romano, S., Baldassarre, M. T., Caivano, D., Scanniello, G., Turhan, B., & Juristo, N. (2018). A longitudinal cohort study on the retainment of test-driven development. Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. ESEM ’18 (pp. 18–11810). New York, NY, USA: ACM. https://doi.org/10.1145/3239235.3240502
George, B., & Williams, L. (2003). An initial investigation of test driven development in industry. Proc. of the ACM SAC 2003. SAC ’03 (pp. 1135–1139). New York, NY, USA: ACM. https://doi.org/10.1145/952532.952753
Ghafari, M., Eggiman, M., & Nierstrasz, O. (2019). Testability first! 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) (pp. 1–6). https://doi.org/10.1109/ESEM.2019.8870170
Ghafari, M., Gross, T., Fucci, D., & Felderer, M. (2020). Why research on test-driven development is inconclusive? Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). ESEM ’20. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3382494.3410687
Guerra, E., & Aniche, M. (2016). Chapter 9 - achieving quality on software design through test-driven development. In I. M. S. A. G. Tekinerdogan (Ed.), Software Quality Assurance (pp. 201–220). Boston: Morgan Kaufmann. https://doi.org/10.1016/B978-0-12-802301-3.00009-0
Hannay, J. E., Dybå, T., Arisholm, E., & Sjøberg, D. I. K. (2009). The effectiveness of pair programming: A meta-analysis. Information and Software Technology, 51, 1110–1122. https://doi.org/10.1016/j.infsof.2009.02.001
IEEE. (1990). Ieee standard glossary of softw. eng. terminology. IEEE Standard Glossary of Softw. Eng. Terminology. New York: IEEE Computer Society Press.
Karac, E. I., Turhan, B., & Juristo, N. (2019). A controlled experiment with novice developers on the impact of task description granularity on software quality in test-driven development. IEEE Transactions on Software Engineering. https://doi.org/10.1109/TSE.2019.2920377
Kazerouni, A. M., Shaffer, C. A., Edwards, S. H., & Servant, F. (2019). Assessing incremental testing practices and their impact on project outcomes. Proceedings of the 50th ACM Technical Symposium on Computer Science Education. SIGCSE ’19 (pp. 407–413). New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3287324.3287366
Krasner, H. (2021). The cost of poor software quality in the us: A 2020 report. Technical Report CISQ-TR-2021-01, Consortium for Information & Software Quality. Accessed 26 Mar 2021. https://www.it-cisq.org/the-cost-of-poor-software-quality-in-the-us-a-2020-report/
L. Salge, C.A., Berente, N. (2016). Pair programming vs. solo programming: What do we know after 15 years of research? In: 2016 49th Hawaii International Conference on System Sciences (HICSS), pp. 5398–5406. https://doi.org/10.1109/HICSS.2016.667
Latorre, R. (2014). Effects of developer experience on learning and applying unit test-driven development. IEEE Transactions on Software Engineering, 40(4), 381–395. https://doi.org/10.1109/TSE.2013.2295827
Lemos, O. A. L., Bajracharya, S., Ossher, J., Masiero, P. C., & Lopes, C. (2011). A test-driven approach to code search and its application to the reuse of auxiliary functionality. Information and Software Technology, 53, 294–306.
Lemos, O. A. L., Ferrari, F. C., Silveira, F. F., & Garcia, A. (2012). Development of auxiliary functions: Should you be agile? an empirical assessment of pair programming and test-first programming. 2012 34th International Conference on Software Engineering (ICSE), (pp. 529–539). https://doi.org/10.1109/ICSE.2012.6227163
Lemos, O. A. L., Silveira, F. F., Ferrari, F. C., & Garcia, A. (2018). The impact of software testing education on code reliability: An empirical assessment. Journal of Systems and Software, 137, 497–511. https://doi.org/10.1016/j.jss.2017.02.042
Madeyski, L. (2010). Test-Driven Development: An Empirical Evaluation of Agile Practice (1st ed.). Berlin: Springer.
Matheny, K. (2015). Why is it that when pair programming produces better code, almost no company practices it? http://qr.ae/RkhnlG. Accessed 07 Jun 2021
Meyer, B. (2014). Agile!: The Good, the Hype and the Ugly (1st ed.). Berlin: Springer.
Montgomery, D. C. (2006). Design and Analysis of Experiments. New York: John Wiley & Sons.
Munir, H., Moayyed, M., & Petersen, K. (2014). Considering rigor and relevance when evaluating test driven development: A systematic review. Information and Software Technology, 56(4), 375–394. https://doi.org/10.1016/j.infsof.2014.01.002
Myers, G. J., Sandler, C., Badgett, T., & Thomas, T. M. (2004). The Art of Software Testing (2nd ed.). New York: John Wiley & Sons.
Nagappan, N., Maximilien, E. M., Bhat, T., & Williams, L. (2008). Realizing quality improvement through test driven development: Results and experiences of four industrial teams. Empirical Software Engineering, 13(3), 289–302.
Pressman, R. S., & Maxim, B. R. (2020). Software Engineering: A Practitioner’s Approach. Berlin: McGraw-Hill Education.
Rafique, Y., & Mišić, V. B. (2013). The effects of test-driven development on external quality and productivity: A meta-analysis. IEEE Transactions on Software Engineering, 39(6), 835–856. https://doi.org/10.1109/TSE.2012.28
Romano, S., Fucci, D., Baldassarre, M. T., Caivano, D., & Scanniello, G. (2019). An empirical assessment on affective reactions of novice developers when applying test-driven development. In X. Franch, T. Männistö, & S. Martínez-Fernández (Eds.), Product-Focused Software Process Improvement (pp. 3–19). Cham: Springer.
Salleh, N., Mendes, E., & Grundy, J. (2010). Empirical studies of pair programming for CS/SE teaching in higher education: A systematic literature review. IEEE Transactions on Software Engineering. https://doi.org/10.1109/TSE.2010.59. PrePrints.
Saltz, J.S., Shamshurin, I. (2017). Does pair programming work in a data science context? an initial case study. In: 2017 IEEE International Conference on Big Data (Big Data), pp. 2348–2354. https://doi.org/10.1109/BigData.2017.8258189
Santos, A., Vegas, S., Dieste, O., Uyaguari, F., Tosun, A., Fucci, D., Turhan, B., Scanniello, G., Romano, S., Karac, I., Kuhrmann, M., Mandić, V., Ramač, R., Pfahl, D., Engblom, C., Kyykka, J., Rungi, K., Palomeque, C., Spisak, J., … Juristo, N. (2021). A family of experiments on test-driven development. Empirical Software Engineering, 26(3), 42. https://doi.org/10.1007/s10664-020-09895-8
Shull, F., Singer, J., & Sjøberg, D. I. K. (2008). Guide to Advanced Empirical Software Engineering. Secaucus, NJ, USA: Springer.
Siegmund, J., Siegmund, N., & Apel, S. (2015). Views on internal and external validity in empirical software engineering. 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering (Vol. 1, pp. 9–19). https://doi.org/10.1109/ICSE.2015.24
Sillitti, A., Succi, G., & Vlasenko, J. (2012). Understanding the impact of pair programming on developers attention: A case study on a large industrial experimentation. 2012 34th International Conference on Software Engineering (ICSE) (pp. 1094–1101). https://doi.org/10.1109/ICSE.2012.6227110
Spence, N. (2011). Apple offers advice on iPhone alarm bug. http://www.macworld.com/article/1155509/dst_bug_iphone.html. Accessed 07 Jun 2021
Sun, W., Marakas, G., & Aguirre-Urreta, M. (2016). The effectiveness of pair programming: Software professionals’ perceptions. IEEE Software, 33(4), 72–79. https://doi.org/10.1109/MS.2015.106
Swamidurai, R., & Umphress, D. (2012). Collaborative-adversarial pair programming. ISRN. Software Engineering. https://doi.org/10.5402/2012/516184
Tosun, A., Dieste, O., Vegas, S., Pfahl, D., Rungi, K., & Juristo, N. (2021). Investigating the impact of development task on external quality in test-driven development: An industry experiment. IEEE Transactions on Software Engineering, 47(11), 2438–2456. https://doi.org/10.1109/TSE.2019.2949811
Trikha, R. (2014). Pair Programming: Yay or Nay? http://www.cybercoders.com/insights/pair-programming-yay-or-nay. Accessed 07 Jun 2021
Weimer, W., Forrest, S., Le Goues, C., & Nguyen, T. (2010). Automatic program repair with evolutionary computation. Communications of the ACM, 53, 109–116.
Williams, L. (2012). What agile teams think of agile principles. Communications of the ACM, 55(4), 71–76. https://doi.org/10.1145/2133806.2133823
Williams, L., & Kessler, R. (2002). Pair Programming Illuminated (p. 71). Boston, MA, USA: Addison-Wesley Longman Publishing Co., Inc. Chap. 15.
Williams, L., Kessler, R. R., Cunningham, W., & Jeffries, R. (2000). Strengthening the case for pair programming. IEEE Software, 17(4), 19–25. https://doi.org/10.1109/52.854064
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2000). Experimentation in Software Engineering: an Introduction. Norwell, MA, USA: Kluwer Academic Publishers.
Xu, F., & Correia, A.-P. (2023). Adopting distributed pair programming as an effective team learning activity: a systematic review. Journal of Computing in Higher Education. https://doi.org/10.1007/s12528-023-09356-3
Zhong, B., & Li, T. (2020). Can pair learning improve students’ troubleshooting performance in robotics education? Journal of Educational Computing Research, 58(1), 220–248.
Zieris, F., & Prechelt, L. (2021). Two elements of pair programming skill. Proceedings of the 43rd International Conference on Software Engineering: New Ideas and Emerging Results. ICSE-NIER ’21 (pp. 51–55). New York: IEEE Press. https://doi.org/10.1109/ICSE-NIER52604.2021.00019
Funding
Partial financial support was received from Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP) and Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq).
Author information
Authors and Affiliations
Contributions
All authors contributed equally to this work (study conception, design, material preparation, data collection and analysis, and writing).
Corresponding author
Ethics declarations
Conflicts of interest
The authors have no relevant financial or non-financial interests to disclose.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Lemos, O., Silveira, F., Ferrari, F. et al. Unraveling the code: an in-depth empirical study on the impact of development practices in auxiliary functions implementation. Software Qual J 32, 1137–1174 (2024). https://doi.org/10.1007/s11219-024-09682-4
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11219-024-09682-4