Software requirements prioritization and selection using linguistic tools and constraint solvers—a controlled experiment | Empirical Software Engineering Skip to main content
Springer Nature Link
Log in

Software requirements prioritization and selection using linguistic tools and constraint solvers—a controlled experiment

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Implementing the entire set of requirements for a software system is often not feasible owing to time and resource limitations. A key driver for successful delivery of any software system is the ability to prioritize the large number of requirements. Prioritization of requirements is a key challenge because current methods are not scalable to handle a realistic number of requirements. Current methods for requirements prioritization in market-driven software development projects are neither sufficient nor proven. A prioritization technique that is more time-efficient, accurate, and easier to implement for large-scale projects than current practices is needed. We address these challenges with a prioritization method that incorporates the use of a linguistic tool and constraint solver. In this paper we propose a method, referred to as SNIPR, for requirements prioritization and selection based on natural language processing and satisfiability modulo theories solvers. We present a controlled experiment in which 40 systems engineers prioritized and selected 20 requirements from a list of 100 using SNIPR and the weighted sum model. Results show that the SNIPR method consumes less time, improves selection accuracy, and is easier to perform than the weighted sum model. These results motivate further research using linguistic tools and constraint solvers for the prioritization of large sets of requirements.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  • Aasem, M., Ramzan, M., & Jaffar, A (2010) Analysis and optimization of software requirements prioritization techniques. International conference on information and emerging technologies (ICETET), IEEE Computer Society, 1–6

  • Abran, A., Moore, J., & Dupuis, R (2001) SWEBOK Guide to the software engineering body of knowledge, IEEE. Ding, W. and Marchionini, G. 1997. A study on video browsing strategies. Technical report. University of Maryland at College Park

  • Afshari A, Mojahed M, Yusuff RM (2010) Simple additive weighting approach to personnel selection problem. Int J Innov Manag Technol 1(5):511–515

    Google Scholar 

  • Ahl, A (2005) An experimental comparison of five prioritization techniques - investigating ease of use, accuracy, and scalability. Master Thesis No. MSE-2005-11, School of Engineering, Blekinge Institute of Technology

  • Aho AV, Hopcroft JE, Ullman JD (1983) Data structures and algorithms. Addison-Wesley, Reading

    MATH  Google Scholar 

  • Aurum A, Wohlin C (2003) The fundamental nature of requirements engineering activities as a decision-making process. Inf Softw Technol 45(14):945–954

    Article  Google Scholar 

  • Avesani, P., Bazzanella, C., Perini, A., & Susi, A (2004) Supporting the requirements prioritization process. A machine learning approach, in proceedings of 16th international conference on software engineering and knowledge engineering (SEKE 2004), pages 306–311, Banff, Alberta, Canada, June 2004. KSI press

  • Avesani, P., Bazzanella, C., Perini, A., & Susi, A (2005) Facing scalability issues in requirements prioritization with machine learning techniques. In proceedings of the 13th IEEE international requirements engineering conference, IEEE computer society, Paris, 297–305

  • Babar, I. B., Ramzan, M., & Ghayyur, S. A. K (2011) Challenges and future trends in software requirements prioritization. International conference on computer networks and information technology 2011, 319–324

  • Bagchi P, Rao RP (1992) Decision making in mergers: an application of the analytic hierarchy process. Manag Decis Econ 13(2):91–99

    Article  Google Scholar 

  • Barney S, Aurum A, Wohlin C (2008) A product management challenge: creating software product value through requirements selection. J Syst Archit 54:576–593

    Article  Google Scholar 

  • Beg, M., Abbas, Q., & Verma, R (2008) An approach for requirement prioritization using B-Tree. First international conference on emerging trends in engineering and technology (ICITET), IEEE computer society, 1216–1221

  • Berander, P (2004) Using students as subjects in requirements prioritization. proceedings of the 2004 international symposium on empirical software engineering (ISESE’04). IEEE Computer Society, 167–176

  • Berander P, Andrews A (2005) Requirements prioritization. In: Aurum A, Wohlin C (eds) Engineering and managing software requirements. Springer Verlag, Berlin, pp 69–94

    Chapter  Google Scholar 

  • Berander P, Svahnberg M (2009) Evaluating two ways of calculating priorities in requirements hierarchies - an experiment on hierarchical cumulative voting. J Syst Softw 82(5):836–850

    Article  Google Scholar 

  • Berander, P., Khan, K. A., & Lehtola, L (2006) Towards a research framework on requirements prioritization. proceedings of the sixth conference on software engineering research and practice in Sweden, 39–48

  • Bjarnason E, Wnuk K, Regnell B (2012) Are you biting off more than you can chew? A case study on causes and effects of overscoping in large-scale software engineering. J Inf Softw Technol 54:1107–1124

    Article  Google Scholar 

  • Carlshamre, P., Sandahl, K., Lindvall, M., Regnell, B., & Natt och Dag, J (2001) An industrial survey of requirements interdependencies in software product release planning. Proceedings of the fifth IEE international symposium on requirements engineering, 84–91

  • Danesh, A., Mortazavi, S., & Danesh, S (2009) Requirements prioritization in on-line banking systems: using value-oriented framework. International conference on computer technology and development (ICCT), IEEE computer society, 158–161

  • Daneva, M., & Herrmann, A (2008) Requirements prioritization based on benefit and cost prediction: a method classification framework. Press 34th Euromicro conference software engineering and advanced applications, IEEE computer society, 240–247

  • Davis AM (2003) The art of requirements triage. IEEE Compute 36(3):42–49

    Article  Google Scholar 

  • Durillo, J., Zhang Y., Alba, E., & Nebro, A (2009) A study of the multi-objective next release problem. IEEE computer society, 49–58

  • Dutertre, B., & De Moura, L (2006) The Yices SMT solver. Web. http://yices.csl.sri.com/tool-paper.pdf. Accessed 5 May 2013

  • Falessi, D., Cantone, G., & Canfora, G (2010) A comprehensive characterization of NLP techniques for identifying equivalent requirements. Proceedings of the 2010 ACM-IEEE international symposium on empirical software engineering and measurement

  • Firesmith D (2004) Prioritizing requirements. J Object Technol 3(8):35–47

    Article  Google Scholar 

  • Greer, D., & Ruhe, G. (2003). Software release planning: an evolutionary and iterative approach. Information and software technology, pp 243–253

  • Greer D, Bustard D, Sunazuka T (1999a) Effecting and measuring risk reduction in software development. NEC J Res Dev 40(3):378–438

    Google Scholar 

  • Greer, D., Bustard, D. W., & Sunazuka, T (1999b) Prioritisation of system changes using cost-benefit and risk assessments. Proceedings IEEE international symposium on requirements engineering, 180–187

  • Hatton, S. (2008). Choosing the right prioritisation method. Proceedings of the 19th Australian conference on software engineering, 517–526

  • Herrmann A, Paech B (2009) Practical challenges of requirements prioritization based on risk estimation. Empir Softw Eng 14:644–674

    Article  Google Scholar 

  • Holbrook EA, Hayes JH, Dekhtyar A, Li W (2013) A study of methods for textual satisfaction assessment. Empir Softw Eng 18:139–176

    Article  Google Scholar 

  • Höst, M., Wohlin, C., & Thelin, T (2005) Experimental context classification: incentives and experience of subjects. Proceedings of the 27th international conference on software engineering (ICSE), 470–478

  • Jantunen S, Lehtola L, Gause DC, Dumdum UR, Barnes RJ (2011) The challenge of release planning. In Software product management (IWSPM), 2011 Fifth International workshop. IEEE, pp 36–45

  • Jung, H. W (1998) Optimizing value and cost in requirements analysis. IEEE Software, July/August, 74–78

  • Karatzas K, Dioudi E, Moussiopoulos N (2003) Identification of major components for integrated urban air quality management and information systems via user requirements prioritisation. Environ Model Softw 18(2):173–178

    Article  Google Scholar 

  • Karlsson, J (1996) Software requirements prioritizing. Proceedings of the 2nd IEEE international conference on requirements engineering, 110–116

  • Karlsson, J., & Ryan, K (1996) Supporting the selection of software requirements. Proceedings of the 8th international workshop on software specification and design, 146–149

  • Karlsson J, Ryan K (1997) A cost-value approach for prioritizing requirements. IEEE Softw 14(5):67–74

    Article  Google Scholar 

  • Karlsson J, Olsson S, Ryan K (1997) Improved practical support for large-scale requirements prioritising. Requir Eng 2:51–60

    Article  Google Scholar 

  • Karlsson J, Wohlin C, Regnell B (1998) An evaluation of methods for prioritizing software requirements. Inf Softw Technol 39:939–947

    Article  Google Scholar 

  • Karlsson, L., Dahlstedt, A. G., Dag. J. N., Regnell, B., & Persson, A (2003) Challenges in market-driven requirements engineering - an industrial interview study. Proceedings of the eighth international workshop on requirements engineering: foundation for software quality, 101–112

  • Karlsson, L., Berander, P., Regnell, B., & Wohlin, C (2004) Requirements prioritisation: an experiment on exhaustive pair-wise comparisons versus planning game partitioning. Proceedings of the 8th international conference on empirical assessment in software engineering, 145–154

  • Karlsson L, Thelin T, Regnell B, Berander P, Wohlin C (2007) Pair-wise comparisons versus planning game partitioning-experiments on requirements prioritisation techniques. Empir Softw Eng 12(1):3–33

    Article  Google Scholar 

  • Khan, K. A (2006) A systematic review of software requirements prioritization (Unpublished MS Thesis). School of Engineering, Blekinge Institute of Technology

  • Khurum M, Gorschek T, Angelis L, Feldt R (2009) A controlled experiment of a method for early requirements triage utilizing product strategies. Requirements engineering: foundation for software quality. Lect Notes Comput Sci 5512:22–36

    Article  Google Scholar 

  • Laurent, P., Cleland-Huang, J., & Duan, C (2007) Towards automated requirements triage. 15th IEEE international requirements engineering conference, 131–140

  • Leffingwell D, Widrig D (2000) Managing software requirements—a unified approach. Addison-Wesley, Upper Saddle River

    Google Scholar 

  • Lehtola L, Kauppinen M (2006) Suitability of requirements prioritization methods for market-driven software product development. Softw Process Improv Pract 11(1):7–19

    Article  Google Scholar 

  • Li C, Akker M, Brinkkemper S, Diepen G (2010) An integrated approach for requirement selection and scheduling in software release planning. Requir Eng 15:375–396

    Article  Google Scholar 

  • Lubars, M., Potts, C., & Richter, C (1993) A review of the state of the practice in requirements modeling. Proceedings of the IEEE international symposium of requirements engineering, 2–14

  • Ma, Q (2009) The effectiveness of requirements prioritization techniques for a medium to large number of requirements: a systematic literature review. school of computing and mathematical sciences, Auckland University of Technology

  • Mack, N., Woodsong, C., MacQueen, K. M., Guest, G., & Namey, E (2005) Qualitative research methods: a data collector’s field guide. Family health international (pp. 6–9).

  • Natt och Dag J, Gervasi V, Brinkkemper S, Regnell B (2005) A linguistic approach to large-scale requirements management. IEEE Softw 22(1):32–39

    Article  Google Scholar 

  • Natt och Dag J, Regnell B, Carlshamre P, Anderson M, Karlsson J (2002) A feasibility study of automated natural language requirements analysis in market-driven development. Requir Eng 7(1):20–33

    Article  MATH  Google Scholar 

  • Natt och Dag J, Thelin T, Regnell B (2006) An experiment on linguistic tool support for consolidation of requirements from multiple sources in market-driven product development. Empir Softw Eng 11:303–329

    Article  Google Scholar 

  • Ngo-The A, Ruhe G (2005) Decision support in requirements engineering. In: Aurum A, Wohlin C (eds) Engineering and managing software requirements. Springer Verlag, Berlin, pp 267–286

    Chapter  Google Scholar 

  • Palma, F., Susi, A., Tonella, P (2011) Using an SMT solver for interactive requirements prioritization. Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on foundations of software engineering ACM, 48–58

  • Perini, A., Ricca, F., Susi, A., & Bazzanella, C (2007) An empirical study to compare the accuracy of AHP and CBRanking techniques for requirements prioritization. Fifth international workshops on comparative evaluation in requirements engineering, 23–35

  • Port, D., Olkov, A., & Menzie, T (2008) Using simulation to investigate requirements prioritization strategies. Proceedings of the 23rd IEEE/ACM international conference on automated software engineering, 268–277

  • Racheva, Z., Daneva, M., & Buglione, L (2008) Supporting the dynamic reprioritization of requirements in agile development of software products. Second international workshop on software product management, 49–58

  • Racheva, Z., Daneva, M., Herrmann, A., & Wieringa, R.J. (2010). A conceptual model and process for client-driven agile requirements prioritization. Fourth international conference on research challenges in information science (RCIS), 287–298

  • Regnell B, Beremark P, Eklundh O (1998) A market-driven requirements engineering process: results from an industrial process improvement programme. Requir Eng 3(2):121–129

    Article  Google Scholar 

  • Regnell B, Runeson P, Thelin T (2000) Are the perspectives really different? – Further experimentation on scenario-based reading of requirements. Empir Softw Eng 5:331–356

    Article  MATH  Google Scholar 

  • Regnell B, Höst M, NattochDag J, Beremark P, Hjelm T (2001) An industrial case study on distributed prioritisation in market-driven requirements engineering for packaged software. Requir Eng 6(1):51–62

    Article  MATH  Google Scholar 

  • Regnell, B., Karlsson, L., & Host, M (2003) An analytical model for requirements selection quality evaluation in product software development. Proceedings of the 11th IEEE international requirements engineering conference, 254–263

  • Regnell, B., Svensson, R., & Wnuk, K (2008a) Can we beat the complexity of very large-scale requirements engineering? Springer-Verlag Berlin Heidelberg, 123–128

  • Regnell B, Svensson RB, Wnuk K (2008b) Requirements engineering: foundation for software quality. Lect Notes Comput Sci 5025(2008):123–128

    Article  Google Scholar 

  • Reifer DJ (2002) How good are agile methods? IEEE Softw 19(4):16–18

    Article  Google Scholar 

  • Ruhe, G., & Greer, D (2003) Quantitative studies in software release planning under risk and resource constraints. Empirical software engineering, 262–270

  • Ruhe G, Saliu MO (2005) The art and science of software release planning. Softw IEEE 22(6):47–53

    Article  Google Scholar 

  • Ruhe, G., Eberlein, A., & Pfahl, D (2002) Quantitative WinWin: a new method for decision support in requirements negotiation. Proceedings of the 14th international conference on software engineering and knowledge engineering, 159–166

  • Saaty TL (1980) The analytic hierarchy process. McGraw-Hill, New York

    MATH  Google Scholar 

  • Saliu OM (2009) Release planning of software systems. VDM Verlag, Saarbrücken

    Google Scholar 

  • Sjøberg DI, Hannay JE, Hansen O, Kampenes VB, Karahasanovic A, Liborg NK, Rekdal AC (2005) A survey of controlled experiments in software engineering. IEEE Trans Softw Eng 31(9):733–753

    Article  Google Scholar 

  • Sommerville I, Sawyer P (1997) Requirements engineering - a good practice guide. Wiley, Chichester

    MATH  Google Scholar 

  • Svahnberg M, Goreschek T, Feldt R, Torkar R, Saleem SB, Shafique MU (2010) A systematic review on strategic release planning models. Inf Softw Technol 52:237–248

    Article  Google Scholar 

  • Svensson, R.B., Gorschek, T., Regnell, B., Torkar, R., Shahrokni, A., Feldt, R., & Aurum, A (2011) Prioritization of quality requirements: state of practice in eleven companies. proceedings of the requirements engineering conference (RE), 2011 19th IEEE International, 69–78

  • Tonella, P., Susi, A., & Palma, F (2010) Using Interactive GA for requirements prioritization. 2nd international symposium on search based software engineering, IEEE Computer society, 57–66

  • Tonella, P., Susi, A., & Palma, F (2012) Interactive requirements prioritization using a genetic algorithm. Information and software technology, 57–66

  • Tonella P, Susi A, Palma F (2013) Interactive requirements prioritization using a genetic algorithm. Inf Softw Technol 55:173–187

    Article  Google Scholar 

  • Triantaphyllou E (2000) Multi-criteria decision making methods. Appl Optim 44:5–21

    Article  Google Scholar 

  • Triantaphyllou, E., & Baig, K (2005) The impact of aggregating benefit and cost criteria in four MCDA methods. IEEE transactions on engineering management, 213–226

  • van den Akker, M., Brinkkemper, S., Diepen, G., & Versendaal, J (2005) Determination of the next release of a software product: an approach using integer linear programming. Proceedings from REFSQ’05: The 11th International workshop on requirements engineering: Foundation for software quality, 119–124

  • van den Akker M, Brinkkemper S, Diepen G, Versendaal J (2008) Software product release planning through optimization and what-if analysis. Inf Softw Technol 50:101–111

    Article  Google Scholar 

  • Wnuk K, Regnell B, Schrewelius C (2009) Architecting and coordinating thousands of requirements – an industrial case study. Requirements engineering: foundation for software quality. Lect Notes Comput Sci 5512:118–123

    Article  Google Scholar 

  • Wnuk K, Regnell B, Berenbach B (2011) Scaling up Requirements Engineering – Exploring the Challenges of Increasing Size and Complexity in Market-Driven Software Development. Requir Eng Found Softw Qual Lect Notes Comput Sci 6606:54–59

    Article  Google Scholar 

  • Yap B, Sim H (2011) Comparisons of various types of normality tests. J Stat Comput Simul 81(12):2141–2155

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgments

The authors would like to thank the participants in this research experiment and Dr. Thomas Mazzuchi for his guidance and approval of the experiment. The George Washington University Office of Human Research is acknowledged for providing institutional oversight of the human subject research activities.

Author information

Authors and Affiliations

  1. Department of Engineering Management & Systems Engineering, The George Washington University, Washington, DC, 20052, USA

    Jason McZara, Shahryar Sarkani, Thomas Holzer & Timothy Eveleigh

Authors
  1. Jason McZara
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shahryar Sarkani
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thomas Holzer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Timothy Eveleigh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jason McZara.

Additional information

Communicated by: Daniel M. Berry

Appendix: Activity Diagrams for the Methods

Appendix: Activity Diagrams for the Methods

Fig. 13
figure 13

Activity diagram for the SNIPR method, detailing the interaction between the user and computer (note the four different activities performed by the user)

Full size image
Fig. 14
figure 14

Activity diagram for the WSM, detailing the interaction between the user and computer

Full size image

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

McZara, J., Sarkani, S., Holzer, T. et al. Software requirements prioritization and selection using linguistic tools and constraint solvers—a controlled experiment. Empir Software Eng 20, 1721–1761 (2015). https://doi.org/10.1007/s10664-014-9334-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-014-9334-8

Keywords

Search

Navigation