Abstract
Software architecture evaluation involves evaluating different architecture design alternatives against multiple quality-attributes. These attributes typically have intrinsic conflicts and must be considered simultaneously in order to reach a final design decision. AHP (Analytic Hierarchy Process), an important decision making technique, has been leveraged to resolve such conflicts. AHP can help provide an overall ranking of design alternatives. However it lacks the capability to explicitly identify the exact tradeoffs being made and the relative size of these tradeoffs. Moreover, the ranking produced can be sensitive such that the smallest change in intermediate priority weights can alter the final order of design alternatives. In this paper, we propose several in-depth analysis techniques applicable to AHP to identify critical tradeoffs and sensitive points in the decision process. We apply our method to an example of a real-world distributed architecture presented in the literature. The results are promising in that they make important decision consequences explicit in terms of key design tradeoffs and the architecture's capability to handle future quality attribute changes. These expose critical decisions which are otherwise too subtle to be detected in standard AHP results.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Al-Naeem, T., Gorton, I., Babar, M.A., Rabhi, F., and Benatallah, B. 2005. A quality-driven systematic approach for architecting distributed software applications, In Proceedings of the 27th International Conference on Software Engineering (ICSE), St. Louis, USA.
Aurum, A. and Wohlin, C. 2003. The fundamental nature of requirements engineering activities as a decision-making process, Information and Software Technology 45(14): 945–954.
Bass, L., Clements, P., and Kazman, R. 2003. Software Architecture in Practice, Addison-Wesley.
Bengtsson, P., Lassing, N., Bosch, J., and Vliet, H.v. 2004. Architecture-level modifiability analysis (ALMA), Journal of Systems and Software 69(1/2): 129–147.
Bosch, J. 2000. Design & Use of Software Architectures: Adopting and Evolving a Product-Line Approach, Addison-Wesley.
Bosch, J. 2004. Software architecture: The next step, In Proceedings of the First European Workshop on Software Architecture (EWSA), pp. 194–199.
Chung, L., Nixon, B.A., Yu, E., and Mylopoluos, J. 2000. Non-Functional Requirements in Software Engineering. Dordrecht, Kluwer Academic Publishers.
Dobrica, L. and Niemela, E. 2002. A survey on software architecture analysis methods, IEEE Transactions on Software Engineering 28(7).
Expert Choice (version 11). 2004. http://www.expertchoice.com/Last accessed on 26th Sept., 2004.
Finan, J.S. and Hurley, W.J. 1996. A note on a method to ensure rank-order consistency in the analytic hierarchy process, International Transactions in Operational Research 3(1): 99–103.
Giorgini, P., Mylopoulos, J., Nicchiarelli, E., and Sebastiani, R. 2004. Formal reasoning techniques for goal models, Journal on Data Semantics 1: 1–20.
Gorton, I. and Haack, J. 2004. Architecting in the Face of Uncertainty: An experience report, In Proceedings of the 26th International Conference on Software Engineering (ICSE), Edinburgh, United Kingdom, pp. 543–551.
Gross, D. and Yu, E.S.K. 2001. From non-functional requirements to design through patterns, Requirement Engineering 6(1): 18–36.
In, H., Boehm, B., Rodgers, T., and Deutsch, M. 2001. Applying WinWin to quality requirements: A case study, In Proceedings of the 23rd International Conference on Software Engineering (ICSE), pp. 555–564.
In, H., Kazman, R., and Olson, D. 2001. From requirements negotiation to software architectural decisions, In Proceedings of the First International Workshop from Software Requirements to Architectures (STRAW'01).
Jennings, D. and Wattam, S. 1998. Decision Making: An Integrated Approach, London, Washington, D.C., Financial Times Pitman Pub.
Karlsson, J., Wohlin, C., and Regnell, B. 1998. An evaluation of methods for prioritizing software requirements, Information and Software Technology 39(14/15): 938–947.
Kazman, R., Asundi, J., and Klein, M. 2001. Quantifying the costs and benefits of architectural decision, In Proceedings of the 23rd International Conference on Software Engineering (ICSE), pp. 297–306.
Kazman, R., Barbacci, M., Klein, M., and Carriere, J. 1999. Experience with performing architecture tradeoff analysis, In Proceedings of the 21st International Conferences on Software Engineering (ICSE'99), pp. 54–63.
Klein, M. and Kazman, R. 1999. Attribute-Based Architectural Styles, Soft Engineering Institute, Carnegie Mellon University. CMU/SEI-99-TR-022.
Kleindorfer, P.R., Kunreuther, H., and Schoemaker, P.J.H. 1993. Decision Sciences: An Integrative Perspective, Cambridge, England; New York, N.Y., Cambridge University Press.
Lindblom, C.E. 1959. The science of muddling through, Public Administrative Review 19: 79–88.
Liu, L. and Yu, E. 2001. From requirements to architectural design - using goals and scenarios, In Proceedings of the First International Workshop from Software Requirements to Architectures (STRAW'01).
Nurmuliani, N., Zowghi, D., and Powell, S. 2004. Analysis of requirements volatility during software development life cycle, In Proceedings of the Software Engineering Conference, 2004. Proceedings. 2004 Australian, pp. 28–37.
Saaty, T.L. 1980. The Analytic Hierarchy Process, New York, McGraw Hill.
Svahnberg, M. and Wohlin, C. 2002. Consensus building when comparing software architectures, In Proceedings of the 4th International Conference on Product Focused Software Process Improvement (PROFES), pp. 436–452.
Svahnberg, M., Wohlin, C., Lundberg, L., and Mattsson, M. 2002. A method for understanding quality attributes in software architecture structures, In Proceedings of the 14th international conference on Software engineering and knowledge engineering (SEKE), pp. 819–826.
Svahnberg, M., Wohlin, C., Lundberg, L., and Mattsson, M. 2003. A quality-driven decision-support method for identifying software architecture candidates, International Journal of Software Engineering and Knowledge Engineering 13(5): 547–573.
Triantaphyllou, E., Kovalerchuk, B., Jr, L.M., and Knapp, G.M. 1997. Determining the most important criteria in maintenance decision making, Journal of Quality in Maintenance Engineering 3(1): 16–28.
Triantaphyllou, E. and Mann, S.H. 1994. Some critical issues in making decisions with pair-wise comparisons, In Proceedings of the Third International Symposium on the AHP, pp. 225–236.
Author information
Authors and Affiliations
Corresponding author
Additional information
Liming Zhu is a PHD candidate in the School of Computer Science and Engineering at University of New South Wales. He is also a member of the Empirical Software Engineering Group at National ICT Australia (NICTA). He obtained his BSc from Dalian University of Technology in China. After moving to Australia, he obtained his MSc in computer science from University of New South Wales. His principle research interests include software architecture evaluation and empirical software engineering.
Aybüke Aurum is a senior lecturer at the School of Information Systems, Technology and Management, University of New South Wales. She received her BSc and MSc in geological engineering, and MEngSc and PhD in computer science. She also works as a visiting researcher in National ICT, Australia (NICTA). Dr. Aurum is one of the editors of “Managing Software Engineering Knowledge”, “Engineering and Managing Software Requirements” and “Value-Based Software Engineering” books. Her research interests include management of software development process, software inspection, requirements engineering, decision making and knowledge management in software development. She is on the editorial boards of Requirements Engineering Journal and Asian Academy Journal of Management.
Ian Gorton is a Senior Researcher at National ICT Australia. Until Match 2004 he was Chief Architect in Information Sciences and Engineering at the US Department of Energy's Pacific Northwest National Laboratory. Previously he has worked at Microsoft and IBM, as well as in other research positions. His interests include software architectures, particularly those for large-scale, high-performance information systems that use commercial off-the-shelf (COTS) middleware technologies. He received a PhD in Computer Science from Sheffield Hallam University.
Dr. Ross Jeffery is Professor of Software Engineering in the School of Computer Science and Engineering at UNSW and Program Leader in Empirical Software Engineering in National ICT Australia Ltd. (NICTA). His current research interests are in software engineering process and product modeling and improvement, electronic process guides and software knowledge management, software quality, software metrics, software technical and management reviews, and software resource modeling and estimation. His research has involved over fifty government and industry organizations over a period of 15 years and has been funded from industry, government and universities. He has co-authored four books and over one hundred and twenty research papers. He has served on the editorial board of the IEEE Transactions on Software Engineering, and the Wiley International Series in Information Systems and he is Associate Editor of the Journal of Empirical Software Engineering. He is a founding member of the International Software Engineering Research Network (ISERN). He was elected Fellow of the Australian Computer Society for his contribution to software engineering research.
Rights and permissions
About this article
Cite this article
Zhu, L., Aurum, A., Gorton, I. et al. Tradeoff and Sensitivity Analysis in Software Architecture Evaluation Using Analytic Hierarchy Process. Software Qual J 13, 357–375 (2005). https://doi.org/10.1007/s11219-005-4251-0
Issue Date:
DOI: https://doi.org/10.1007/s11219-005-4251-0