{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,6,12]],"date-time":"2024-06-12T13:58:54Z","timestamp":1718200734957},"reference-count":65,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,3,10]],"date-time":"2021-03-10T00:00:00Z","timestamp":1615334400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,3,10]],"date-time":"2021-03-10T00:00:00Z","timestamp":1615334400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Software Qual J"],"published-print":{"date-parts":[[2022,3]]},"abstract":"Abstract<\/jats:title>Test automation brings the potential to reduce costs and human effort, but several aspects of software testing remain challenging to automate. One such example is automated performance testing to find performance breaking points. Current approaches to tackle automated generation of performance test cases mainly involve using source code or system model analysis or use-case-based techniques. However, source code and system models might not always be available at testing time. On the other hand, if the optimal performance testing policy for the intended objective in a testing process instead could be learned by the testing system, then test automation without advanced performance models could be possible. Furthermore, the learned policy could later be reused for similar software systems under test, thus leading to higher test efficiency. We propose SaFReL, a self-adaptive fuzzy reinforcement learning-based performance testing framework. SaFReL learns the optimal policy to generate performance test cases through an initial learning phase, then reuses it during a transfer learning phase, while keeping the learning running and updating the policy in the long term. Through multiple experiments in a simulated performance testing setup, we demonstrate that our approach generates the target performance test cases for different programs more efficiently than a typical testing process and performs adaptively without access to source code and performance models.<\/jats:p>","DOI":"10.1007\/s11219-020-09532-z","type":"journal-article","created":{"date-parts":[[2021,3,10]],"date-time":"2021-03-10T08:03:33Z","timestamp":1615363413000},"page":"127-159","update-policy":"http:\/\/dx.doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["An autonomous performance testing framework using self-adaptive fuzzy reinforcement learning"],"prefix":"10.1007","volume":"30","author":[{"ORCID":"http:\/\/orcid.org\/0000-0003-3354-1463","authenticated-orcid":false,"given":"Mahshid Helali","family":"Moghadam","sequence":"first","affiliation":[]},{"given":"Mehrdad","family":"Saadatmand","sequence":"additional","affiliation":[]},{"given":"Markus","family":"Borg","sequence":"additional","affiliation":[]},{"given":"Markus","family":"Bohlin","sequence":"additional","affiliation":[]},{"given":"Bj\u00f6rn","family":"Lisper","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2021,3,10]]},"reference":[{"issue":"4","key":"9532_CR1","doi-asserted-by":"publisher","first-page":"801","DOI":"10.1007\/s11219-011-9135-x","volume":"19","author":"A Adamoli","year":"2011","unstructured":"Adamoli, A., Zaparanuks, D., Jovic, M., Hauswirth, M. (2011). Automated gui performance testing. Software Quality Journal,\u00a019(4),\u00a0801\u2013839.","journal-title":"Software Quality Journal"},{"key":"9532_CR2","doi-asserted-by":"crossref","unstructured":"Ahmad, T., Ashraf, A., Truscan, D., Porres, I. (2019). Exploratory performance testing using reinforcement learning. In\u00a02019 45th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)(pp. 156-163). IEEE.","DOI":"10.1109\/SEAA.2019.00032"},{"key":"9532_CR3","doi-asserted-by":"crossref","unstructured":"Apte, V., Viswanath, T. V. S., Gawali, D., Kommireddy, A., Gupta, A. (2017). AutoPerf: Automated load testing and resource usage profiling of multi-tier internet applications. In\u00a0Proceedings of the 8th ACM\/SPEC on International Conference on Performance Engineering\u00a0(pp. 115-126).","DOI":"10.1145\/3030207.3030222"},{"key":"9532_CR4","doi-asserted-by":"crossref","unstructured":"Avritzer, A., Duarte, F. P., Leao, R. M. M., e Silva, Ed. S., Cohen, M., Costello, D. (2008). Reliability estimation for large distributed software systems. In Cascon, Citeseer (p. 12).","DOI":"10.1145\/1463788.1463804"},{"key":"9532_CR5","doi-asserted-by":"crossref","unstructured":"Ayala-Rivera, V., Kaczmarski, M., Murphy, J., Darisa, A., Portillo-Dominguez, AO. (2018). One size does not fit all: In-test workload adaptation for performance testing of enterprise applications. In Proceedings of the 2018 ACM\/SPEC International Conference on Performance Engineering, ACM\u00a0(pp. 211\u2013222).","DOI":"10.1145\/3184407.3184418"},{"key":"9532_CR6","doi-asserted-by":"publisher","first-page":"6974","DOI":"10.1109\/ACCESS.2016.2615181","volume":"4","author":"ZB Babovic","year":"2016","unstructured":"Babovic, Z. B., Protic, J., Milutinovic, V. (2016). Web performance evaluation for internet of things applications. IEEE Access,\u00a04,\u00a06974\u20136992.","journal-title":"IEEE Access<\/i>"},{"key":"9532_CR7","doi-asserted-by":"crossref","unstructured":"Briand, LC., Labiche, Y., Shousha, M. (2005). Stress testing real-time systems with genetic algorithms. In Proceedings of the 7th annual conference on Genetic and evolutionary computation, ACM\u00a0(pp. 1021\u20131028).","DOI":"10.1145\/1068009.1068183"},{"key":"9532_CR8","unstructured":"Brunnert, A., van Hoorn, A., Willnecker, F., Danciu, A., Hasselbring, W., Heger, C., et al. (2015). Performance oriented devops: A research agenda. arXiv preprint.\u00a0arXiv:150804752"},{"issue":"3","key":"9532_CR9","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1145\/1541880.1541882","volume":"41","author":"V Chandola","year":"2009","unstructured":"Chandola, V., Banerjee, A., Kumar, V. (2009). Anomaly detection: A survey. ACM computing surveys (CSUR), 41(3), 15.","journal-title":"ACM computing surveys (CSUR)<\/i>"},{"key":"9532_CR10","unstructured":"Chung, L., Nixon, B. A., Yu, E., Mylopoulos, J. (2012). Non-functional requirements in software engineering. Springer Science & Business Media, 5."},{"key":"9532_CR11","doi-asserted-by":"crossref","unstructured":"Cortellessa, V., Di Marco, A., Inverardi, P. (2011). Model-based software performance analysis. Springer Science & Business Media.","DOI":"10.1007\/978-3-642-13621-4"},{"key":"9532_CR12","unstructured":"Costa, LT., Czekster, RM., de Oliveira, FM., Rodrigues, EDM., da Silveira, MB., Zorzo, AF. (2012). Generating Performance Test Scripts and Scenarios Based on Abstract Intermediate Models. In\u00a0SEKE,\u00a0(pp. 112-117)."},{"key":"9532_CR13","doi-asserted-by":"publisher","first-page":"94","DOI":"10.1145\/974043.974059","volume":"29","author":"G Denaro","year":"2004","unstructured":"Denaro, G., Polini, A., Emmerich, W. (2004). Early performance testing of dis- tributed software applications. ACM SIGSOFT Software Engineering Notes,\u00a029,\u00a094\u2013103.","journal-title":"ACM SIGSOFT Software Engineering Notes<\/i>"},{"key":"9532_CR14","doi-asserted-by":"crossref","unstructured":"Di\u00a0Penta, M., Canfora, G., Esposito, G., Mazza, V., Bruno, M. (2007). Search-based testing of service level agreements. In\u00a0Proceedings of the 9th annual conference on Genetic and evolutionary computation, ACM (pp. 1090\u20131097).","DOI":"10.1145\/1276958.1277174"},{"key":"9532_CR15","doi-asserted-by":"crossref","unstructured":"Draheim, D., Grundy, J., Hosking, J., Lutteroth, C., Weber, G. (2006). Realistic load testing of web applications. In Conference on Software Maintenance and Reengineering (CSMR\u201906), IEEE\u00a0(p. 11).","DOI":"10.1109\/CSMR.2006.43"},{"key":"9532_CR16","doi-asserted-by":"crossref","unstructured":"Ferme, V., & Pautasso, C. (2017). Towards holistic continuous software performance assessment. In\u00a0Proceedings of the 8th ACM\/SPEC on International Conference on Performance Engineering Companion, ACM\u00a0(pp. 159\u2013164).","DOI":"10.1145\/3053600.3053636"},{"key":"9532_CR17","doi-asserted-by":"crossref","unstructured":"Ferme, V., & Pautasso, C. (2018). A declarative approach for performance tests execution in continuous software development environments. In Proceedings of the 2018 ACM\/SPEC International Conference on Performance Engineering, ACM\u00a0(pp. 261\u2013272).","DOI":"10.1145\/3184407.3184417"},{"key":"9532_CR18","unstructured":"Fowler, K. (2009).\u00a0Mission-critical and safety-critical systems handbook: Design and development for embedded applications. Newnes."},{"key":"9532_CR19","doi-asserted-by":"crossref","unstructured":"Garousi, V. (2008).\u00a0Empirical analysis of a genetic algorithm-based stress test technique. In Proceedings of the 10th annual conference on Genetic and evolutionary computation, ACM\u00a0(pp.\u00a01743\u20131750).","DOI":"10.1145\/1389095.1389433"},{"issue":"6","key":"9532_CR20","doi-asserted-by":"publisher","first-page":"778","DOI":"10.1109\/TSE.2010.5","volume":"36","author":"V Garousi","year":"2010","unstructured":"Garousi, V. (2010). A genetic algorithm-based stress test requirements generator tool and its empirical evaluation. IEEE Transactions on Software Engineering, 36(6), 778\u2013797.","journal-title":"IEEE Transactions on Software Engineering<\/i>"},{"issue":"2","key":"9532_CR21","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1016\/j.jss.2007.05.037","volume":"81","author":"V Garousi","year":"2008","unstructured":"Garousi, V., Briand, L. C., Labiche, Y. (2008). Traffic-aware stress testing of distributed real-time systems based on uml models using genetic algorithms. Journal of Systems and Software,\u00a081(2), 161\u2013185.","journal-title":"Journal of Systems and Software<\/i>"},{"key":"9532_CR22","doi-asserted-by":"crossref","unstructured":"Glinz, M. (2007).\u00a0On non-functional requirements. In 15th IEEE International Requirements Engineering Conference (RE 2007), IEEE\u00a0(pp.\u00a021\u201326).","DOI":"10.1109\/RE.2007.45"},{"key":"9532_CR23","doi-asserted-by":"crossref","unstructured":"Grechanik, M., Fu, C., Xie, Q. (2012).\u00a0Automatically finding performance problems with feedback-directed learning software testing. In 2012 34th International Conference on Software Engineering (ICSE), IEEE\u00a0(pp.\u00a0156\u2013166).","DOI":"10.1109\/ICSE.2012.6227197"},{"key":"9532_CR24","unstructured":"Gregg, B. (2013).\u00a0Systems performance: enterprise and the cloud. Pearson Education."},{"key":"9532_CR25","doi-asserted-by":"crossref","unstructured":"Grinshpan, L. (2012). Solving enterprise applications performance puzzles: queuing models to the rescue. John Wiley & Sons.","DOI":"10.1002\/9781118161920"},{"key":"9532_CR26","doi-asserted-by":"crossref","unstructured":"Gu, Y., &\u00a0Ge, Y. (2009).\u00a0Search-based performance testing of applications with composite services. In 2009 International Conference on Web Information Systems and Mining, IEEE\u00a0(pp.\u00a0320\u2013324).","DOI":"10.1109\/WISM.2009.73"},{"key":"9532_CR27","doi-asserted-by":"crossref","unstructured":"Harchol-Balter, M. (2013). Performance modeling and design of computer systems: queueing theory in action. Cambridge University Press.","DOI":"10.1017\/CBO9781139226424"},{"issue":"4","key":"9532_CR28","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1109\/MS.2009.197","volume":"27","author":"J Hill","year":"2009","unstructured":"Hill, J., Schmidt, D., Edmondson, J., Gokhale, A. (2009). Tools for continuously evaluating distributed system qualities. IEEE software,\u00a027(4), 65\u201371.","journal-title":"IEEE software<\/i>"},{"issue":"1","key":"9532_CR29","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1145\/2791120","volume":"48","author":"O Ibidunmoye","year":"2015","unstructured":"Ibidunmoye, O., Hernandez-Rodriguez, F., Elmroth, E. (2015). Performance anomaly detection and bottleneck identification. ACM Computing Surveys (CSUR),\u00a048(1), 4.","journal-title":"ACM Computing Surveys (CSUR)"},{"key":"9532_CR30","doi-asserted-by":"crossref","unstructured":"Ibidunmoye, O., Moghadam, M. H., Lakew, E. B., Elmroth, E. (2017). Adaptive service performance control using cooperative fuzzy reinforcement learning in vir- tualized environments. In Proceedings of the10th International Conference on Utility and Cloud Computing, ACM\u00a0(pp. 19\u201328).","DOI":"10.1145\/3147213.3147225"},{"key":"9532_CR31","unstructured":"ISO 25000 (2019). ISO\/IEC 25010 - System and software quality models. Available at\u00a0https:\/\/iso25000.com\/index.php\/en\/iso-25000-standards\/iso-25010.\u00a0Retrieved July, 2019."},{"key":"9532_CR32","doi-asserted-by":"crossref","unstructured":"Jamshidi, P., Sharifloo, A., Pahl, C., Arabnejad, H., Metzger, A., Estrada, G. (2016). Fuzzy self-learning controllers for elasticity management in dynamic cloud architectures. In 2016 12th International ACM SIGSOFT Conference on Quality of Software Architectures (QoSA), IEEE\u00a0(pp. 70\u201379).","DOI":"10.1109\/QoSA.2016.13"},{"issue":"3","key":"9532_CR33","doi-asserted-by":"publisher","first-page":"567","DOI":"10.1007\/s10922-014-9307-7","volume":"23","author":"B Jennings","year":"2015","unstructured":"Jennings, B., & Stadler, R. (2015). Resource management in clouds: Survey and research challenges. Journal of Network and Systems Management,\u00a023(3), 567\u2013619.","journal-title":"Journal of Network and Systems Management<\/i>"},{"issue":"11","key":"9532_CR34","doi-asserted-by":"publisher","first-page":"1091","DOI":"10.1109\/TSE.2015.2445340","volume":"41","author":"ZM Jiang","year":"2015","unstructured":"Jiang, Z. M., & Hassan, A. E. (2015). A survey on load testing of large-scale software systems. IEEE, Transactions on Software Engineering,\u00a041(11), 1091\u20131118.","journal-title":"IEEE, Transactions on Software Engineering<\/i>"},{"key":"9532_CR35","doi-asserted-by":"crossref","unstructured":"Jindal, A., Podolskiy, V., Gerndt, M. (2019). Performance modeling for cloud microservice applications. In Proceedings of the 2019 ACM\/SPEC International Conference on Performance Engineering, ACM\u00a0(pp. 25\u201332).","DOI":"10.1145\/3297663.3310309"},{"key":"9532_CR36","unstructured":"Kant, K., & Srinivasan, M. (1992). Introduction to computer system performance evaluation. McGraw-Hill College."},{"issue":"3","key":"9532_CR37","doi-asserted-by":"publisher","first-page":"2265","DOI":"10.1007\/s10270-018-0662-9","volume":"18","author":"S Kolesnikov","year":"2019","unstructured":"Kolesnikov, S., Siegmund, N., Kastner, C., Grebhahn, A., Apel, S. (2019). Tradeoffs in modeling performance of highly configurable software systems. Software & Systems Modeling,\u00a018(3), 2265\u20132283.","journal-title":"Software & Systems Modeling<\/i>"},{"key":"9532_CR38","doi-asserted-by":"crossref","unstructured":"Koo, J., Saumya, C., Kulkarni, M., Bagchi, S. (2019). Pyse: Automatic worst-case test generation by reinforcement learning. In 2019 12th IEEE Conference on Software Testing, Validation and Verification\u00a0(ICST), IEEE\u00a0(pp. 136\u2013147).","DOI":"10.1109\/ICST.2019.00023"},{"issue":"1","key":"9532_CR39","doi-asserted-by":"publisher","first-page":"2925","DOI":"10.4249\/scholarpedia.2925","volume":"3","author":"LI Kuncheva","year":"2008","unstructured":"Kuncheva, L. I. (2008). Fuzzy classifiers. Scholarpedia,\u00a03(1), 2925.","journal-title":"Fuzzy classifiers. Scholarpedia<\/i>"},{"key":"9532_CR40","doi-asserted-by":"crossref","unstructured":"Lutteroth, C., & Weber, G. (2008). Modeling a realistic workload for performance testing. In 2008 12th International IEEE Enterprise Distributed Object Computing Conference, IEEE\u00a0(pp. 149\u2013158).","DOI":"10.1109\/EDOC.2008.40"},{"key":"9532_CR41","doi-asserted-by":"crossref","unstructured":"Maddodi, G., Jansen, S., de\u00a0Jong, R. (2018). Generating workload for erp applications through end-user organization categorization using high level business operation data. In Proceedings of the 2018 ACM\/SPEC International Conference on Performance Engineering, ACM\u00a0(pp. 200\u2013210).","DOI":"10.1145\/3184407.3184432"},{"key":"9532_CR42","doi-asserted-by":"crossref","unstructured":"Malik, H., Jiang, Z. M., Adams, B., Hassan, A. E., Flora, P., Hamann, G. (2010). Automatic comparison of load tests to support the performance analysis of large enterprise systems. In 2010 14th European conference on software maintenance and reengineering, IEEE\u00a0(pp. 222\u2013231).","DOI":"10.1109\/CSMR.2010.39"},{"key":"9532_CR43","doi-asserted-by":"crossref","unstructured":"Malik, H., Hemmati, H., Hassan, A. E. (2013). Automatic detection of performance deviations in the load testing of large scale systems. In Proceedings of the 2013 International Conference on Software Engineering, IEEE Press\u00a0(pp. 1012\u20131021).","DOI":"10.1109\/ICSE.2013.6606651"},{"key":"9532_CR44","unstructured":"MathWorks (2019). Fuzzy Inference Process. Retrieved from\u00a0https:\/\/www.mathworks.com\/help\/fuzzy\/fuzzy-inference-process.html"},{"key":"9532_CR45","unstructured":"Menasc\u2019e, DA. (2002). Load testing, benchmarking, and application performance management for the web. In Int. CMG Conference\u00a0(pp. 271\u2013282)."},{"key":"9532_CR46","doi-asserted-by":"crossref","unstructured":"Michael, N., Ramannavar, N., Shen, Y., Patil, S., Sung, J. L. (2017). Cloudperf: A performance test framework for distributed and dynamic multi-tenant environ- ments. In Proceedings of the 8th ACM\/SPEC on International Conference on Performance Engineering, ACM\u00a0(pp. 189\u2013200).","DOI":"10.1145\/3030207.3044530"},{"key":"9532_CR47","doi-asserted-by":"crossref","unstructured":"Moghadam, M. H., Saadatmand, M., Borg, M., Bohlin, M., Lisper, B. (2018). Adaptive runtime response time control in plc-based real-time systems using rein- forcement learning. In 2018 IEEE\/ACM 13th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS), IEEE\u00a0(pp. 217\u2013223).","DOI":"10.1145\/3194133.3194153"},{"key":"9532_CR48","unstructured":"Moghadam, M. H., Saadatmand, M., Borg, M., Bohlin, M., Lisper, B. (2019). Machine learning to guide performance testing: An autonomous test framework. In 2019 IEEE International Conference on Software Testing,\u00a0Verification and Validation Workshops (ICSTW), IEEE\u00a0(pp. 164\u2013167)."},{"key":"9532_CR49","doi-asserted-by":"publisher","first-page":"8835","DOI":"10.1109\/ACCESS.2017.2704444","volume":"5","author":"R Morabito","year":"2017","unstructured":"Morabito, R. (2017). Virtualization on internet of things edge devices with con- tainer technologies: a performance evaluation. IEEE Access,\u00a05, 8835\u20138850.","journal-title":"IEEE Access<\/i>"},{"key":"9532_CR50","unstructured":"NS8 (2018). Did You Know A Slow Webpage Can Cost You 7% of Your Sales. Available at\u00a0https:\/\/www.ns8.com\/en\/ns8u\/did-you-know\/a-slowwebpage-can-cost-you-7-percent-of-your-sales. Retrieved July 2019"},{"key":"9532_CR51","doi-asserted-by":"crossref","unstructured":"Schulz, H., Okanovi\u2019c, D., van\u00a0Hoorn, A., Ferme, V., Pautasso, C. (2019). Behavior- driven load testing using contextual knowledge-approach and experiences. In Proceedings of the 2019 ACM\/SPEC International Conference on Performance Engineering, ACM\u00a0(pp. 265\u2013272).","DOI":"10.1145\/3297663.3309674"},{"key":"9532_CR52","doi-asserted-by":"crossref","unstructured":"Shams, M., Krishnamurthy, D., Far, B. (2006). A model-based approach for testing the performance of web applications. In Proceedings of the 3rd international workshop on Software quality assurance, ACM\u00a0(pp. 54\u201361).","DOI":"10.1145\/1188895.1188909"},{"key":"9532_CR53","unstructured":"da\u00a0Silveira, MB., Rodrigues, EdM., Zorzo, AF., Costa, LT., Vieira, HV., de\u00a0Oliveira, FM. (2011). Generation of scripts for performance testing based on uml models. In SEKE\u00a0(pp. 258\u2013263)."},{"key":"9532_CR54","unstructured":"Sutton, RS., & Barto, AG. (2018). Reinforcement learning: An introduction. MIT press."},{"key":"9532_CR55","doi-asserted-by":"crossref","unstructured":"Syer, MD., Adams, B., Hassan, AE. (2011). Identifying performance deviations in thread pools. In 2011 27th IEEE International Conference on Software Maintenance (ICSM), IEEE\u00a0(pp. 83\u201392).","DOI":"10.1109\/ICSM.2011.6080775"},{"key":"9532_CR56","doi-asserted-by":"crossref","unstructured":"Taheri, J., Zomaya, AY., Kassler, A. (2016). vmbbthrpred: A black-box throughput predictor for virtual machines in cloud environments. In European Conference on Service-Oriented and Cloud Computing\u00a0(pp. 18\u201333). Springer.","DOI":"10.1007\/978-3-319-44482-6_2"},{"issue":"3","key":"9532_CR57","doi-asserted-by":"publisher","first-page":"145","DOI":"10.1504\/IJBDI.2016.078400","volume":"3","author":"T Veni","year":"2016","unstructured":"Veni, T., Bhanu, S. M. S. (2016). Auto-scale: automatic scaling of virtualised re- sources using neuro-fuzzy reinforcement learning approach. International Journal of Big Data Intelligence,\u00a03(3), 145\u2013153.","journal-title":"International Journal of Big Data Intelligence<\/i>"},{"key":"9532_CR58","unstructured":"Venkataraman, S., Yang, Z., Franklin, M., Recht, B., Stoica, I. (2016). Ernest: ef- ficient performance prediction for large-scale advanced analytics. In 13th USENIX Symposium on Networked Systems Design and Implementation, NSDI (16)\u00a0(pp. 363\u2013378)."},{"issue":"2","key":"9532_CR59","doi-asserted-by":"publisher","first-page":"443","DOI":"10.1007\/s10270-016-0566-5","volume":"17","author":"C Vogele","year":"2018","unstructured":"Vogele, C., van Hoorn, A., Schulz, E., Hasselbring, W., Krcmar, H. (2018). Wessbas: extraction of probabilistic workload specifications for load testing and per- formance prediction-a model-driven approach for session-based application systems. Software & Systems Modeling,\u00a017(2), 443\u2013477.","journal-title":"Software & Systems Modeling<\/i>"},{"key":"9532_CR60","doi-asserted-by":"crossref","unstructured":"Walter, J., van\u00a0Hoorn, A., Koziolek, H., Okanovic, D., Kounev, S. (2016). Asking what?, automating the how?: The vision of declarative performance engi- neering. In Proceedings of the 7th ACM\/SPEC on International Conference on Performance Engineering, ACM\u00a0(pp. 91\u201394).","DOI":"10.1145\/2851553.2858662"},{"issue":"12","key":"9532_CR61","doi-asserted-by":"publisher","first-page":"1147","DOI":"10.1109\/32.888628","volume":"26","author":"EJ Weyuker","year":"2000","unstructured":"Weyuker, E. J., & Vokolos, F. I. (2000). Experience with performance testing of software systems: issues, an approach, and case study. IEEE transactions on software engineering,\u00a026(12), 1147\u20131156.","journal-title":"IEEE transactions on software engineering<\/i>"},{"key":"9532_CR62","doi-asserted-by":"publisher","first-page":"201","DOI":"10.1145\/226295.226318","volume":"21","author":"CSD Yang","year":"1996","unstructured":"Yang, C. S. D., & Pollock, L. L. (1996). Towards a structural load testing tool. ACM SIGSOFT Software Engineering Notes, ACM, 21, 201\u2013208.","journal-title":"ACM SIGSOFT Software Engineering Notes, ACM<\/i>"},{"key":"9532_CR63","doi-asserted-by":"crossref","unstructured":"Zhang, J., & Cheung, SC. (2002). Automated test case generation for the stress testing of multimedia systems.\u00a0Software: Practice and Experience,\u00a032(15), 1411\u20131435.","DOI":"10.1002\/spe.487"},{"key":"9532_CR64","doi-asserted-by":"crossref","unstructured":"Zhang, P., Elbaum, S., Dwyer, M. B. (2011). Automatic generation of load tests. In Proceedings of the 2011 26th IEEE\/ACM International Conference on Automated Software Engineering, IEEE Computer Society\u00a0(pp. 43\u201352).","DOI":"10.1109\/ASE.2011.6100093"},{"key":"9532_CR65","doi-asserted-by":"crossref","unstructured":"Zhang, P., Elbaum, S., Dwyer, M. B. (2012). Compositional load test generation for software pipelines. In Proceedings of the 2012 International Symposium on Software Testing and Analysis, ACM\u00a0(pp. 89\u201399).","DOI":"10.1145\/2338965.2336764"}],"container-title":["Software Quality Journal"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11219-020-09532-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s11219-020-09532-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s11219-020-09532-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,20]],"date-time":"2022-12-20T23:40:10Z","timestamp":1671579610000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s11219-020-09532-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,10]]},"references-count":65,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,3]]}},"alternative-id":["9532"],"URL":"https:\/\/doi.org\/10.1007\/s11219-020-09532-z","relation":{},"ISSN":["0963-9314","1573-1367"],"issn-type":[{"value":"0963-9314","type":"print"},{"value":"1573-1367","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,3,10]]},"assertion":[{"value":"24 September 2020","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 March 2021","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}