{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,4,19]],"date-time":"2025-04-19T16:44:04Z","timestamp":1745081044964,"version":"3.37.3"},"reference-count":66,"publisher":"Wiley","issue":"11","license":[{"start":{"date-parts":[[2018,7,3]],"date-time":"2018-07-03T00:00:00Z","timestamp":1530576000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"funder":[{"name":"Swedish Government and Industrial Excellence Center Embedded Applications Software Engineering (EASE)"}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["J Software Evolu Process"],"published-print":{"date-parts":[[2018,11]]},"abstract":"Abstract<\/jats:title>DevOps as a new way of thinking for software development and operations has received much attention in the industry, while it has not been thoroughly investigated in academia yet. The objective of this study is to characterize DevOps by exploring its central components in terms of principles, practices and their relations to the principles, challenges of DevOps adoption, and benefits reported in the peer\u2010reviewed literature. As a key objective, we also aim to realize the relations between DevOps practices and benefits in a systematic manner. A systematic literature review was conducted. Also, we used the concept of benefits dependency network to synthesize the findings, in particular, to specify dependencies between DevOps practices and link the practices to benefits. We found that in many cases, DevOps characteristics, ie, principles, practices, benefits, and challenges, were not sufficiently defined in detail in the peer\u2010reviewed literature. In addition, only a few empirical studies are available, which can be attributed to the nascency of DevOps research. Also, an initial version of the DevOps benefits dependency network has been derived. The definition of DevOps principles and practices should be emphasized given the novelty of the concept. Further empirical studies are needed to improve the benefits dependency network presented in this study.<\/jats:p>","DOI":"10.1002\/smr.1957","type":"journal-article","created":{"date-parts":[[2018,7,3]],"date-time":"2018-07-03T09:06:15Z","timestamp":1530608775000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":36,"title":["Towards a benefits dependency network for DevOps based on a systematic literature review"],"prefix":"10.1002","volume":"30","author":[{"given":"Ramtin","family":"Jabbari","sequence":"first","affiliation":[{"name":"Blekinge Institute of Technology Karlskrona Sweden"}]},{"given":"Nauman","family":"bin Ali","sequence":"additional","affiliation":[{"name":"Blekinge Institute of Technology Karlskrona Sweden"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1532-8223","authenticated-orcid":false,"given":"Kai","family":"Petersen","sequence":"additional","affiliation":[{"name":"Blekinge Institute of Technology Karlskrona Sweden"},{"name":"Flensburg University of Applied Sciences Flensburg Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2300-068X","authenticated-orcid":false,"given":"Binish","family":"Tanveer","sequence":"additional","affiliation":[{"name":"Fraunhofer Institute for Experimental Software Engineering Kaiserslautern Germany"}]}],"member":"311","published-online":{"date-parts":[[2018,7,3]]},"reference":[{"key":"e_1_2_10_2_1","doi-asserted-by":"publisher","DOI":"10.1142\/S0218194006003002"},{"key":"e_1_2_10_3_1","doi-asserted-by":"publisher","DOI":"10.1002\/smr.1560"},{"key":"e_1_2_10_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2012.01.008"},{"key":"e_1_2_10_5_1","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-29263-2"},{"key":"e_1_2_10_6_1","unstructured":"TollW.Top DevOps conferences in 2016: 49 events for DevOps pros.2015. ProfitBricks Blog https:\/\/blog.profitbricks.com\/."},{"key":"e_1_2_10_7_1","unstructured":"de\u00a0Fran\u00e7aBBN JeronimoHJr TravassosGH.Characterizing DevOps by hearing multiple voices. In: Proceedings of the 30th Brazilian Symposium on Software Engineering SBES 2016; September 19\u201023 2016;Maring\u00e1 Brazil.53\u201062."},{"key":"e_1_2_10_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2524713.2524721"},{"key":"e_1_2_10_9_1","unstructured":"KitchenBA ChartersS.Guidelines for performing systematic literature reviews in software engineering. EBSE Technical Report EBSE\u20102007\u201001 United Kingdom \u00a0 Keele University Keele Staffs;2007."},{"key":"e_1_2_10_10_1","unstructured":"BadewiAA ShehabE PeppardJ.Benefit realisation modelling for ERP systems using system dynamics. In: Advances in Manufacturing Technology XXVII\u2010Proceedings of the 11th International Conference on Manufacturing Research (ICMR2013) Cranfield University;2013;Cranfield Bedford UK.225\u2010235."},{"issue":"1","key":"e_1_2_10_11_1","first-page":"1","article-title":"Managing the realization of business benefits from it investments","volume":"6","author":"Peppard J","year":"2007","journal-title":"MIS Q Executive"},{"key":"e_1_2_10_12_1","doi-asserted-by":"crossref","unstructured":"AliNB.Is effectiveness sufficient to choose an intervention?: considering resource use in empirical software engineering. In: Proceedings of the 10th ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement ESEM 2016; September 8\u20109 2016;Ciudad Real Spain.54:1\u201054:6.","DOI":"10.1145\/2961111.2962631"},{"key":"#cr-split#-e_1_2_10_13_1.1","doi-asserted-by":"crossref","unstructured":"ErichF AmritC DanevaM.Cooperation between information system development and operations: A literature review. In: 2014 ACM\u2010IEEE International Symposium on Empirical Software Engineering and Measurement ESEM '14","DOI":"10.1145\/2652524.2652598"},{"key":"#cr-split#-e_1_2_10_13_1.2","unstructured":"September 18\u201019 2014;Torino Italy.69:1."},{"key":"e_1_2_10_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-18612-2_14"},{"key":"e_1_2_10_15_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2009.06.041"},{"key":"e_1_2_10_16_1","doi-asserted-by":"crossref","unstructured":"AliNB PetersenK.Evaluating strategies for study selection in systematic literature studies. In: Proceedings of the 8th ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement ESEM '14;2014;New York NY USA.45:1\u201045:4.","DOI":"10.1145\/2652524.2652557"},{"key":"e_1_2_10_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2015.03.007"},{"key":"e_1_2_10_18_1","doi-asserted-by":"crossref","unstructured":"FarrohaBS FarrohaDL.A framework for managing mission needs compliance and trust in the devops environment. In: Military Communications Conference (MILCOM) 2014.Tempa FL USA:IEEE;2014:288\u2010293.","DOI":"10.1109\/MILCOM.2014.54"},{"key":"e_1_2_10_19_1","doi-asserted-by":"crossref","unstructured":"McCarthyMA HergerLM KhanSM BelgodereBM.Composable devops: automated ontology based devops maturity analysis. In: 2015 IEEE International Conference on Services Computing SCC 2015; June 27\u2010July 2 2015;New York City NY USA.600\u2010607.","DOI":"10.1109\/SCC.2015.87"},{"key":"e_1_2_10_20_1","doi-asserted-by":"crossref","unstructured":"VirmaniM.Understanding DevOps bridging the gap from continuous integration to continuous delivery. In: 2015 Fifth International Conference on Innovative Computing Technology (INTECH).Galicia Spain;2015:78\u201082.","DOI":"10.1109\/INTECH.2015.7173368"},{"key":"e_1_2_10_21_1","doi-asserted-by":"crossref","unstructured":"BruneoD FritzT Keidar\u2010BarnerS et al.Cloudwave: Where adaptive cloud management meets DevOps. In: IEEE Symposium on Computers and Communications ISCC 2014; June 23\u201026 2014;Funchal Madeira Portugal.1\u20106.","DOI":"10.1109\/ISCC.2014.6912638"},{"key":"e_1_2_10_22_1","doi-asserted-by":"crossref","unstructured":"CoisCA YankelJ ConnellA.Modern devops: Optimizing software development through effective system interactions. In: 2014 IEEE International Professional Communication Conference IPCC 2014; October 13\u201015 2014;Pittsburgh PA USA.1\u20107.","DOI":"10.1109\/IPCC.2014.7020388"},{"key":"e_1_2_10_23_1","doi-asserted-by":"crossref","unstructured":"FerryN ChauvelF SongH SolbergA.Continous deployment of multi\u2010cloud systems. In: Proceedings of the 1st International Workshop on Quality\u2010Aware DevOps QUDOS 2015; September 1 2015;Bergamo Italy.27\u201028.","DOI":"10.1145\/2804371.2804377"},{"key":"e_1_2_10_24_1","doi-asserted-by":"crossref","unstructured":"KilamoT Lepp\u00e4nenM MikkonenT.The social developer: now then and tomorrow. In: Proceedings of the 7th International Workshop on Social Software Engineering SSE 2015;2015;Bergamo Italy.41\u201048.","DOI":"10.1145\/2804381.2804388"},{"key":"e_1_2_10_25_1","doi-asserted-by":"crossref","unstructured":"KruscheS AlperowitzL.Introduction of continuous delivery in multi\u2010customer project courses. In: Companion Proceedings of 36th International Conference on Software Engineering ICSE'14; May 31\u2010June 07 2014;Hyderabad India.335\u2010343.","DOI":"10.1145\/2591062.2591163"},{"key":"e_1_2_10_26_1","doi-asserted-by":"crossref","unstructured":"N\u00e9methF SteinertR KreugerP Sk\u00f6ldstr\u00f6mP.Roles of DevOps tools in an automated dynamic service creation architecture. In: IFIP\/IEEE International Symposium on Integrated Network Management IM 2015; May 11\u201015 2015;Ottawa ON Canada.1153\u20101154.","DOI":"10.1109\/INM.2015.7140455"},{"key":"e_1_2_10_27_1","doi-asserted-by":"crossref","unstructured":"ParkS ChaB KimJ.Preparing and inter\u2010connecting hyper\u2010converged smartx boxes for iot\u2010cloud testbed. In: 29th IEEE International Conference on Advanced Information Networking and Applications AINA 2015; March 24\u201027 2015;Gwangju South Korea.695\u2010697.","DOI":"10.1109\/AINA.2015.255"},{"key":"e_1_2_10_28_1","doi-asserted-by":"crossref","unstructured":"SegallI Tzoref\u2010BrillR.Feedback\u2010driven combinatorial test design and execution. In: Proceedings of the 8th ACM International Systems and Storage Conference SYSTOR 2015; May 26\u201028 2015;Haifa Israel.12:1\u201012:6.","DOI":"10.1145\/2757667.2757677"},{"key":"e_1_2_10_29_1","unstructured":"WettingerJ BehrendtM BinzT et al.Integrating configuration management with model\u2010driven cloud management based on TOSCA. In: CLOSER 2013 \u2010 Proceedings of the 3rd International Conference on Cloud Computing and Services Science; May 8\u201010 2013;Aachen Germany.437\u2010446."},{"key":"e_1_2_10_30_1","doi-asserted-by":"crossref","unstructured":"WettingerJ Breitenb\u00fccherU LeymannF.Compensation\u2010based vs. convergent deployment automation for services operated in the cloud. In: Proceedings of Service\u2010Oriented Computing \u2010 12th International Conference ICSOC 2014; November 3\u20106 2014;Paris France.336\u2010350.","DOI":"10.1007\/978-3-662-45391-9_23"},{"key":"e_1_2_10_31_1","unstructured":"AustelP ChenH DubeP et al.A PaaS for composite analytics solutions. In: Proceedings of the 12th ACM International Conference on Computing Frontiers CF'15; May 18\u201021 2015;Ischia Italy.51:1\u201051:8."},{"key":"e_1_2_10_32_1","doi-asserted-by":"crossref","unstructured":"DyckA PennersR LichterH.Towards definitions for release engineering and DevOps. In: 3rd IEEE\/ACM International Workshop on Release Engineering RELENG 2015; May 19 2015;Florence Italy.3.","DOI":"10.1109\/RELENG.2015.10"},{"key":"e_1_2_10_33_1","doi-asserted-by":"crossref","unstructured":"HosonoS HeJ LiuX LiL HuangH YoshinoS.Fast development platforms and methods for cloud applications. In: 2011 IEEE Asia\u2010Pacific Services Computing Conference APSCC 2011; December 12\u201015 2011;Jeju Korea (South).94\u2010101.","DOI":"10.1109\/APSCC.2011.75"},{"key":"e_1_2_10_34_1","doi-asserted-by":"crossref","unstructured":"KimJ MeirosuC PapafiliI et al.Service provider DevOps for large scale modern network services. In: IFIP\/IEEE International Symposium on Integrated Network Management IM 2015; May 11\u201015 2015;Ottawa ON Canada.1391\u20101397.","DOI":"10.1109\/INM.2015.7140502"},{"key":"e_1_2_10_35_1","doi-asserted-by":"crossref","unstructured":"WettingerJ AndrikopoulosV LeymannF.Automated capturing and systematic usage of DevOps knowledge for cloud applications. In: 2015 IEEE International Conference on Cloud Engineering IC2E 2015; March 9\u201013 2015;Tempe AZ USA.60\u201065.","DOI":"10.1109\/IC2E.2015.23"},{"key":"e_1_2_10_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2015.23"},{"key":"e_1_2_10_37_1","doi-asserted-by":"crossref","unstructured":"FitzgeraldB StolK\u2010J.Continuous software engineering and beyond: trends and challenges. In: 1st International Workshop on Rapid Continuous Software Engineering RCoSE 2014; June 3 2014;Hyderabad India.1\u20109.","DOI":"10.1145\/2593812.2593813"},{"key":"e_1_2_10_38_1","doi-asserted-by":"crossref","unstructured":"GohilK AlapatiN JoglekarS.Towards behavior driven operations (BDOps). In: 3rd International Conference on Advances in Recent Technologies in Communication and Computing (ARTCom 2011);2011Quingdao China.179\u2010183","DOI":"10.1049\/ic.2011.0095"},{"key":"e_1_2_10_39_1","doi-asserted-by":"crossref","unstructured":"GottesheimW.Challenges benefits and best practices of performance focused DevOps. In: Proceedings of the 4th International Workshop on Large\u2010Scale Testing LT'15; February 1 2015;Austin TX USA.3.","DOI":"10.1145\/2693182.2693187"},{"key":"e_1_2_10_40_1","doi-asserted-by":"crossref","unstructured":"MurphyS GallantS GaughanC DiegoM.U.S. army modeling and simulation executable architecture deployment cloud virtualization strategy. In: 12th IEEE\/ACM International Symposium on Cluster Cloud and Grid Computing CCGrid 2012; May 13\u201016 2012;Ottawa Canada.880\u2010885.","DOI":"10.1109\/CCGrid.2012.89"},{"key":"e_1_2_10_41_1","doi-asserted-by":"crossref","unstructured":"ScheunerJ CitoJ LeitnerP GallHC.Cloud workbench: benchmarking iaas providers based on infrastructure\u2010as\u2010code. In: Proceedings of the 24th International Conference on World Wide Web Companion WWW 2015 \u2010 Companion Volume; May 18\u201022 2015;Florence Italy.239\u2010242.","DOI":"10.1145\/2740908.2742833"},{"key":"e_1_2_10_42_1","doi-asserted-by":"crossref","unstructured":"UstinovaT JamshidiP.Modelling multi\u2010tier enterprise applications behaviour with design of experiments technique. In: Proceedings of the 1st International Workshop on Quality\u2010Aware DevOps QUDOS 2015; September 1 2015;Bergamo Italy.13\u201018.","DOI":"10.1145\/2804371.2804374"},{"key":"e_1_2_10_43_1","doi-asserted-by":"crossref","unstructured":"BangSK ChungS ChohY DupuisM.A grounded theory analysis of modern web applications: knowledge skills and abilities for devops. In: Proceedings of the 2nd Annual Conference on Research in Information Technology RIIT '13;2013;New York NY USA.61\u201062.","DOI":"10.1145\/2512209.2512229"},{"key":"#cr-split#-e_1_2_10_44_1.1","doi-asserted-by":"crossref","unstructured":"P\u00e9rezJF WangW CasaleG.Towards a devops approach for software quality engineering. In: Proceedings of the 2015 Workshop on Challenges in Performance Methods for Software Development WOSP\u2010C'15","DOI":"10.1145\/2693561.2693564"},{"key":"#cr-split#-e_1_2_10_44_1.2","unstructured":"January 31 2015;Austin TX USA.5\u201010."},{"key":"e_1_2_10_45_1","doi-asserted-by":"crossref","unstructured":"MiglierinaM.Application deployment and management in the cloud. In: 16th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing SYNASC 2014; September 22\u201025 2014;Timisoara Romania.422\u2010428.","DOI":"10.1109\/SYNASC.2014.63"},{"key":"e_1_2_10_46_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2015.07.017"},{"key":"e_1_2_10_47_1","doi-asserted-by":"publisher","DOI":"10.1504\/IJCSE.2012.049753"},{"key":"#cr-split#-e_1_2_10_48_1.1","doi-asserted-by":"crossref","unstructured":"JayaramKR.Towards explicitly elastic programming frameworks. In: 37th IEEE\/ACM International Conference on Software Engineering ICSE 2015 Vol.\u00a02","DOI":"10.1109\/ICSE.2015.207"},{"key":"#cr-split#-e_1_2_10_48_1.2","unstructured":"May 16\u201024 2015;Florence Italy.619\u2010622."},{"key":"e_1_2_10_49_1","unstructured":"WettingerJ Breitenb\u00fccherU LeymannF.Devopslang \u2010 bridging the gap between development and operations. In: Proceedings of Service\u2010oriented and Cloud Computing \u2010 Third European Conference ESOCC 2014; September 2\u20104 2014;Manchester UK.108\u2010122."},{"key":"e_1_2_10_50_1","doi-asserted-by":"crossref","unstructured":"OlszewskaM Wald\u00e9nMA.Devops meets formal modelling in high\u2010criticality complex systems. In: Proceedings of the 1st International Workshop on Quality\u2010Aware DevOps QUDOS 2015; September 1 2015;Bergamo Italy.7\u201012.","DOI":"10.1145\/2804371.2804373"},{"key":"e_1_2_10_51_1","doi-asserted-by":"crossref","unstructured":"StillwellM CoutinhoJGF.A devops approach to integration of software components in an EU research project. In: Proceedings of the 1st International Workshop on Quality\u2010Aware DevOps QUDOS 2015; September 1 2015;Bergamo Italy.1\u20106.","DOI":"10.1145\/2804371.2804372"},{"key":"e_1_2_10_52_1","doi-asserted-by":"crossref","unstructured":"de\u00a0BayserM AzevedoLG CerqueiraRFG.Researchops: The case for DevOps in scientific applications. In: IFIP\/IEEE International Symposium on Integrated Network Management IM 2015; May 11\u201015 2015;Ottawa ON Canada.1398\u20101404.","DOI":"10.1109\/INM.2015.7140503"},{"key":"e_1_2_10_53_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2015.06.063"},{"key":"e_1_2_10_54_1","doi-asserted-by":"crossref","unstructured":"HussainiSW.Strengthening harmonization of development (dev) and operations (ops) silos in it environment through systems approach. In: 2014 IEEE 17th International Conference on Intelligent Transportation Systems (ITSC);2014;Barcelona Spain.178\u2010183.","DOI":"10.1109\/ITSC.2014.6957687"},{"key":"e_1_2_10_55_1","doi-asserted-by":"crossref","unstructured":"BoschettiM BaglioV RuiuP TerzoO.A cloud automation platform for flexibility in applications and resources provisioning. In: Ninth International Conference on Complex Intelligent and Software Intensive Systems CISIS 2015; July 8\u201010 2015;Santa Catarina Brazil.204\u2010208.","DOI":"10.1109\/CISIS.2015.29"},{"key":"e_1_2_10_56_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2011.01.004"},{"volume-title":"Guide to the Software Engineering Body of Knowledge: 2004 Version","year":"2004","author":"Abran A","key":"e_1_2_10_57_1"},{"key":"e_1_2_10_58_1","unstructured":"Principles behind the agile manifesto 2017.http:\/\/agilemanifesto.org\/principles.html."},{"key":"e_1_2_10_59_1","doi-asserted-by":"crossref","unstructured":"JabbariR binAliN PetersenK TanveerB.What is DevOps? A systematic mapping study on definitions and practices. In: XP 2016 First International Workshop on Emerging Trends in DevOps and Infrastructure;2016;Edinburgh Scotland UK.12.","DOI":"10.1145\/2962695.2962707"},{"key":"e_1_2_10_60_1","unstructured":"A view of the DevOps cycle 2017.http:\/\/readwrite.com\/2014\/01\/01\/three-reasons-your-startup-needs-devops-or-else."},{"key":"e_1_2_10_61_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2015.27"},{"key":"e_1_2_10_62_1","doi-asserted-by":"crossref","unstructured":"AustelP ChenH MikalsenTA et al.Continuous delivery of composite solutions: a case for collaborative software defined paas environments. In: Proceedings of the 2nd International Workshop on Software\u2010Defined Ecosystems Bigsystem 2015; June 16 2015;Portland Oregon USA.3\u20106.","DOI":"10.1145\/2756594.2756595"},{"key":"e_1_2_10_63_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2013.07.010"},{"key":"e_1_2_10_64_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2013.04.076"}],"container-title":["Journal of Software: Evolution and Process"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1002%2Fsmr.1957","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/smr.1957","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,9,15]],"date-time":"2023-09-15T03:35:34Z","timestamp":1694748934000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1002\/smr.1957"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,7,3]]},"references-count":66,"journal-issue":{"issue":"11","published-print":{"date-parts":[[2018,11]]}},"alternative-id":["10.1002\/smr.1957"],"URL":"https:\/\/doi.org\/10.1002\/smr.1957","archive":["Portico"],"relation":{},"ISSN":["2047-7473","2047-7481"],"issn-type":[{"type":"print","value":"2047-7473"},{"type":"electronic","value":"2047-7481"}],"subject":[],"published":{"date-parts":[[2018,7,3]]},"assertion":[{"value":"2016-12-29","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-04-25","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-07-03","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}