{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,12,30]],"date-time":"2024-12-30T19:14:27Z","timestamp":1735586067274,"version":"3.28.0"},"publisher-location":"New York, NY, USA","reference-count":29,"publisher":"ACM","funder":[{"DOI":"10.13039\/501100002347","name":"Bundesministerium f\u00fcr Bildung und Forschung","doi-asserted-by":"publisher","award":["ATHENE"],"id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003495","name":"Hessisches Ministerium f\u00fcr Wissenschaft und Kunst","doi-asserted-by":"publisher","award":["ATHENE"],"id":[{"id":"10.13039\/501100003495","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,7,10]]},"DOI":"10.1145\/3579856.3582830","type":"proceedings-article","created":{"date-parts":[[2023,7,5]],"date-time":"2023-07-05T14:52:13Z","timestamp":1688568733000},"page":"387-400","update-policy":"http:\/\/dx.doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Benchmarking the Benchmarks"],"prefix":"10.1145","author":[{"ORCID":"http:\/\/orcid.org\/0000-0002-3806-0522","authenticated-orcid":false,"given":"Marc","family":"Miltenberger","sequence":"first","affiliation":[{"name":"Secure Software Engineering, Fraunhofer SIT, Germany and ATHENE - National Research Center for Applied Cybersecurity, Germany"}]},{"ORCID":"http:\/\/orcid.org\/0000-0002-5807-9431","authenticated-orcid":false,"given":"Steven","family":"Arzt","sequence":"additional","affiliation":[{"name":"Secure Software Engineering, Fraunhofer SIT, Germany and ATHENE - National Research Center for Applied Cybersecurity, Germany"}]},{"ORCID":"http:\/\/orcid.org\/0000-0003-4422-8579","authenticated-orcid":false,"given":"Philipp","family":"Holzinger","sequence":"additional","affiliation":[{"name":"Secure Software Engineering, Fraunhofer SIT, Germany and ATHENE - National Research Center for Applied Cybersecurity, Germany"}]},{"ORCID":"http:\/\/orcid.org\/0000-0002-5162-3334","authenticated-orcid":false,"given":"Julius","family":"N\u00e4umann","sequence":"additional","affiliation":[{"name":"Fraunhofer SIT, Germany and ATHENE - National Research Center for Applied Cybersecurity, Germany"}]}],"member":"320","published-online":{"date-parts":[[2023,7,10]]},"reference":[{"key":"#cr-split#-e_1_3_2_1_1_1.1","doi-asserted-by":"crossref","unstructured":"Sharmin Afrose Sazzadur Rahaman and Danfeng Yao. 2019. CryptoAPI-Bench: A Comprehensive Benchmark on Java Cryptographic API Misuses. In 2019 IEEE Cybersecurity Development (SecDev). 49-61. https:\/\/doi.org\/10.1109\/SecDev.2019.00017 10.1109\/SecDev.2019.00017","DOI":"10.1109\/SecDev.2019.00017"},{"key":"#cr-split#-e_1_3_2_1_1_1.2","doi-asserted-by":"crossref","unstructured":"Sharmin Afrose Sazzadur Rahaman and Danfeng Yao. 2019. CryptoAPI-Bench: A Comprehensive Benchmark on Java Cryptographic API Misuses. In 2019 IEEE Cybersecurity Development (SecDev). 49-61. https:\/\/doi.org\/10.1109\/SecDev.2019.00017","DOI":"10.1109\/SecDev.2019.00017"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10270-004-0058-x"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2388936.2388960"},{"key":"e_1_3_2_1_4_1","volume-title":"2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS). 328\u2013336","author":"Artho Cyrille","year":"2021","unstructured":"Cyrille Artho , Adam Benali , and Rudolf Ramler . 2021 . Test Benchmarks: Which One Now and in Future? . In 2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS). 328\u2013336 . https:\/\/doi.org\/10.1109\/QRS54544.2021.00044 10.1109\/QRS54544.2021.00044 Cyrille Artho, Adam Benali, and Rudolf Ramler. 2021. Test Benchmarks: Which One Now and in Future?. In 2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS). 328\u2013336. https:\/\/doi.org\/10.1109\/QRS54544.2021.00044"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2594291.2594299"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3475960.3475985"},{"key":"e_1_3_2_1_7_1","first-page":"10","article-title":"The DaCapo Benchmarks","volume":"41","author":"Blackburn M.","year":"2006","unstructured":"Stephen\u00a0 M. Blackburn , Robin Garner , Chris Hoffmann , Asjad\u00a0 M. Khang , Kathryn\u00a0 S. McKinley , Rotem Bentzur , Amer Diwan , Daniel Feinberg , Daniel Frampton , Samuel\u00a0 Z. Guyer , Martin Hirzel , Antony Hosking , Maria Jump , Han Lee , J.\u00a0Eliot\u00a0 B. Moss , Aashish Phansalkar , Darko Stefanovi\u0107 , Thomas VanDrunen , Daniel von Dincklage , and Ben Wiedermann . 2006 . The DaCapo Benchmarks : Java Benchmarking Development and Analysis. SIGPLAN Not. 41 , 10 (oct 2006), 169\u2013190. https:\/\/doi.org\/10.1145\/1167515.1167488 10.1145\/1167515.1167488 Stephen\u00a0M. Blackburn, Robin Garner, Chris Hoffmann, Asjad\u00a0M. Khang, Kathryn\u00a0S. McKinley, Rotem Bentzur, Amer Diwan, Daniel Feinberg, Daniel Frampton, Samuel\u00a0Z. Guyer, Martin Hirzel, Antony Hosking, Maria Jump, Han Lee, J.\u00a0Eliot\u00a0B. Moss, Aashish Phansalkar, Darko Stefanovi\u0107, Thomas VanDrunen, Daniel von Dincklage, and Ben Wiedermann. 2006. The DaCapo Benchmarks: Java Benchmarking Development and Analysis. SIGPLAN Not. 41, 10 (oct 2006), 169\u2013190. https:\/\/doi.org\/10.1145\/1167515.1167488","journal-title":"Java Benchmarking Development and Analysis. SIGPLAN Not."},{"key":"e_1_3_2_1_8_1","volume-title":"Information Security Practice and Experience: 15th International Conference, ISPEC 2019, Kuala Lumpur, Malaysia, November 26\u201328, 2019, Proceedings, Vol.\u00a011879","author":"Borne Isabelle","year":"2019","unstructured":"Isabelle Borne . 2019 . Designing a Code Vulnerability Meta-scanner . In Information Security Practice and Experience: 15th International Conference, ISPEC 2019, Kuala Lumpur, Malaysia, November 26\u201328, 2019, Proceedings, Vol.\u00a011879 . Springer Nature, 194. Isabelle Borne. 2019. Designing a Code Vulnerability Meta-scanner. In Information Security Practice and Experience: 15th International Conference, ISPEC 2019, Kuala Lumpur, Malaysia, November 26\u201328, 2019, Proceedings, Vol.\u00a011879. Springer Nature, 194."},{"key":"#cr-split#-e_1_3_2_1_9_1.1","doi-asserted-by":"crossref","unstructured":"Gaojian Hao Feng Li Wei Huo Qing Sun Wei Wang Xinhua Li and Wei Zou. 2019. Constructing Benchmarks for Supporting Explainable Evaluations of Static Application Security Testing Tools. In 2019 International Symposium on Theoretical Aspects of Software Engineering (TASE). 65-72. https:\/\/doi.org\/10.1109\/TASE.2019.00-18 10.1109\/TASE.2019.00-18","DOI":"10.1109\/TASE.2019.00-18"},{"key":"#cr-split#-e_1_3_2_1_9_1.2","doi-asserted-by":"crossref","unstructured":"Gaojian Hao Feng Li Wei Huo Qing Sun Wei Wang Xinhua Li and Wei Zou. 2019. Constructing Benchmarks for Supporting Explainable Evaluations of Static Application Security Testing Tools. In 2019 International Symposium on Theoretical Aspects of Software Engineering (TASE). 65-72. https:\/\/doi.org\/10.1109\/TASE.2019.00-18","DOI":"10.1109\/TASE.2019.00-18"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2022.111423"},{"key":"e_1_3_2_1_11_1","volume-title":"The Juliet 1.1 C\/C++ and Java Test Suite. 45 (2012-10-01 00:10:00","author":"Jr Frederick\u00a0Boland","year":"2012","unstructured":"Frederick\u00a0Boland Jr . and Paul Black . 2012. The Juliet 1.1 C\/C++ and Java Test Suite. 45 (2012-10-01 00:10:00 2012 ). https:\/\/doi.org\/10.1109\/MC.2012.345 10.1109\/MC.2012.345 Frederick\u00a0Boland Jr. and Paul Black. 2012. The Juliet 1.1 C\/C++ and Java Test Suite. 45 (2012-10-01 00:10:00 2012). https:\/\/doi.org\/10.1109\/MC.2012.345"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2020.04.217"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/ARES.2013.59"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3319008.3319011"},{"key":"e_1_3_2_1_15_1","volume-title":"Static-Based Test Case Dynamic Generation for SQLIVs Detection. In 2015 10th International Conference on Broadband and Wireless Computing, Communication and Applications (BWCCA). 173\u2013177","author":"Li Ling","year":"2015","unstructured":"Ling Li , Junxin Qi , Nan Liu , Lifang Han , and Baojiang Cui . 2015 . Static-Based Test Case Dynamic Generation for SQLIVs Detection. In 2015 10th International Conference on Broadband and Wireless Computing, Communication and Applications (BWCCA). 173\u2013177 . https:\/\/doi.org\/10.1109\/BWCCA.2015.59 10.1109\/BWCCA.2015.59 Ling Li, Junxin Qi, Nan Liu, Lifang Han, and Baojiang Cui. 2015. Static-Based Test Case Dynamic Generation for SQLIVs Detection. In 2015 10th International Conference on Broadband and Wireless Computing, Communication and Applications (BWCCA). 173\u2013177. https:\/\/doi.org\/10.1109\/BWCCA.2015.59"},{"key":"e_1_3_2_1_16_1","volume-title":"Benchmark Requirements for Assessing Software Security Vulnerability Testing Tools. In 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), Vol.\u00a001","author":"Parizi Reza","year":"2018","unstructured":"Reza M.\u00a0 Parizi , Kai Qian , Hossain Shahriar , Fan Wu , and Lixin Tao . 2018 . Benchmark Requirements for Assessing Software Security Vulnerability Testing Tools. In 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), Vol.\u00a001 . 825\u2013826. https:\/\/doi.org\/10.1109\/COMPSAC.2018.00139 10.1109\/COMPSAC.2018.00139 Reza M.\u00a0Parizi, Kai Qian, Hossain Shahriar, Fan Wu, and Lixin Tao. 2018. Benchmark Requirements for Assessing Software Security Vulnerability Testing Tools. In 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), Vol.\u00a001. 825\u2013826. https:\/\/doi.org\/10.1109\/COMPSAC.2018.00139"},{"key":"e_1_3_2_1_17_1","volume-title":"BenchPress: Analyzing Android App Vulnerability Benchmark Suites. In 2019 34th IEEE\/ACM International Conference on Automated Software Engineering Workshop (ASEW). 13\u201318","author":"Mitra Joydeep","year":"2019","unstructured":"Joydeep Mitra , Venkatesh-Prasad Ranganath , and Aditya Narkar . 2019 . BenchPress: Analyzing Android App Vulnerability Benchmark Suites. In 2019 34th IEEE\/ACM International Conference on Automated Software Engineering Workshop (ASEW). 13\u201318 . https:\/\/doi.org\/10.1109\/ASEW.2019.00020 10.1109\/ASEW.2019.00020 Joydeep Mitra, Venkatesh-Prasad Ranganath, and Aditya Narkar. 2019. BenchPress: Analyzing Android App Vulnerability Benchmark Suites. In 2019 34th IEEE\/ACM International Conference on Automated Software Engineering Workshop (ASEW). 13\u201318. https:\/\/doi.org\/10.1109\/ASEW.2019.00020"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1002\/spe.2346"},{"key":"e_1_3_2_1_20_1","volume-title":"2021 IEEE 21st International Conference on Software Quality, Reliability and Security Companion (QRS-C). IEEE, 60\u201366","author":"Qian Meiyuan","year":"2021","unstructured":"Meiyuan Qian , Jun Luo , Yu Ge , Chen Sun , Xiuting Ge , and Wanmin Huang . 2021 . Semantic-based false alarm detection approach via machine learning . In 2021 IEEE 21st International Conference on Software Quality, Reliability and Security Companion (QRS-C). IEEE, 60\u201366 . Meiyuan Qian, Jun Luo, Yu Ge, Chen Sun, Xiuting Ge, and Wanmin Huang. 2021. Semantic-based false alarm detection approach via machine learning. In 2021 IEEE 21st International Conference on Software Quality, Reliability and Security Companion (QRS-C). IEEE, 60\u201366."},{"key":"e_1_3_2_1_21_1","volume-title":"2019 34th IEEE\/ACM International Conference on Automated Software Engineering Workshop (ASEW). 114\u2013121","author":"Saccente Nicholas","year":"2019","unstructured":"Nicholas Saccente , Josh Dehlinger , Lin Deng , Suranjan Chakraborty , and Yin Xiong . 2019 . Project Achilles: A Prototype Tool for Static Method-Level Vulnerability Detection of Java Source Code Using a Recurrent Neural Network . In 2019 34th IEEE\/ACM International Conference on Automated Software Engineering Workshop (ASEW). 114\u2013121 . https:\/\/doi.org\/10.1109\/ASEW.2019.00040 10.1109\/ASEW.2019.00040 Nicholas Saccente, Josh Dehlinger, Lin Deng, Suranjan Chakraborty, and Yin Xiong. 2019. Project Achilles: A Prototype Tool for Static Method-Level Vulnerability Detection of Java Source Code Using a Recurrent Neural Network. In 2019 34th IEEE\/ACM International Conference on Automated Software Engineering Workshop (ASEW). 114\u2013121. https:\/\/doi.org\/10.1109\/ASEW.2019.00040"},{"key":"e_1_3_2_1_22_1","volume-title":"VANDALIR: Vulnerability Analyses Based on Datalog and LLVM-IR. In International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment. Springer, 96\u2013115","author":"Schilling Joschua","year":"2022","unstructured":"Joschua Schilling and Tilo M\u00fcller . 2022 . VANDALIR: Vulnerability Analyses Based on Datalog and LLVM-IR. In International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment. Springer, 96\u2013115 . Joschua Schilling and Tilo M\u00fcller. 2022. VANDALIR: Vulnerability Analyses Based on Datalog and LLVM-IR. In International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment. Springer, 96\u2013115."},{"key":"e_1_3_2_1_23_1","unstructured":"Michael Schlichtig Anna Wickert Stefan Kr\u00fcger Eric Bodden and Mira Mezini. 2022. CamBench \u2013 Cryptographic API Misuse Detection Tool Benchmark Suite. Michael Schlichtig Anna Wickert Stefan Kr\u00fcger Eric Bodden and Mira Mezini. 2022. CamBench \u2013 Cryptographic API Misuse Detection Tool Benchmark Suite."},{"key":"#cr-split#-e_1_3_2_1_24_1.1","doi-asserted-by":"crossref","unstructured":"Felix Schuckert Hanno Langweg and Basel Katt. 2022. Systematic Generation of XSS and SQLi Vulnerabilities in PHP as Test Cases for Static Code Analysis. 261-268. https:\/\/doi.org\/10.1109\/ICSTW55395.2022.00053 10.1109\/ICSTW55395.2022.00053","DOI":"10.1109\/ICSTW55395.2022.00053"},{"key":"#cr-split#-e_1_3_2_1_24_1.2","doi-asserted-by":"crossref","unstructured":"Felix Schuckert Hanno Langweg and Basel Katt. 2022. Systematic Generation of XSS and SQLi Vulnerabilities in PHP as Test Cases for Static Code Analysis. 261-268. https:\/\/doi.org\/10.1109\/ICSTW55395.2022.00053","DOI":"10.1109\/ICSTW55395.2022.00053"},{"key":"e_1_3_2_1_25_1","volume-title":"SoK: Benchmarking Flaws in Systems Security. In 2019 IEEE European Symposium on Security and Privacy (EuroS&P). 310\u2013325","author":"van\u00a0der Kouwe Erik","year":"2019","unstructured":"Erik van\u00a0der Kouwe , Gernot Heiser , Dennis Andriesse , Herbert Bos , and Cristiano Giuffrida . 2019 . SoK: Benchmarking Flaws in Systems Security. In 2019 IEEE European Symposium on Security and Privacy (EuroS&P). 310\u2013325 . https:\/\/doi.org\/10.1109\/EuroSP.2019.00031 10.1109\/EuroSP.2019.00031 Erik van\u00a0der Kouwe, Gernot Heiser, Dennis Andriesse, Herbert Bos, and Cristiano Giuffrida. 2019. SoK: Benchmarking Flaws in Systems Security. In 2019 IEEE European Symposium on Security and Privacy (EuroS&P). 310\u2013325. https:\/\/doi.org\/10.1109\/EuroSP.2019.00031"},{"key":"e_1_3_2_1_26_1","volume-title":"2019 IEEE 17th International Conference on Software Engineering Research, Management and Applications (SERA). IEEE, 90\u201393","author":"Visalli Nicholas","year":"2019","unstructured":"Nicholas Visalli , Lin Deng , Amro Al-Suwaida , Zachary Brown , Manish Joshi , and Bingyang Wei . 2019 . Towards automated security vulnerability and software defect localization . In 2019 IEEE 17th International Conference on Software Engineering Research, Management and Applications (SERA). IEEE, 90\u201393 . Nicholas Visalli, Lin Deng, Amro Al-Suwaida, Zachary Brown, Manish Joshi, and Bingyang Wei. 2019. Towards automated security vulnerability and software defect localization. In 2019 IEEE 17th International Conference on Software Engineering Research, Management and Applications (SERA). IEEE, 90\u201393."},{"key":"e_1_3_2_1_27_1","volume-title":"2014 11th International Conference on Security and Cryptography (SECRYPT). 1\u20139.","author":"Wagner Andreas","year":"2014","unstructured":"Andreas Wagner and Johannes Sametinger . 2014 . Using the Juliet Test Suite to compare static security scanners . In 2014 11th International Conference on Security and Cryptography (SECRYPT). 1\u20139. Andreas Wagner and Johannes Sametinger. 2014. Using the Juliet Test Suite to compare static security scanners. In 2014 11th International Conference on Security and Cryptography (SECRYPT). 1\u20139."}],"event":{"name":"ASIA CCS '23: ACM ASIA Conference on Computer and Communications Security","sponsor":["SIGSAC ACM Special Interest Group on Security, Audit, and Control"],"location":"Melbourne VIC Australia","acronym":"ASIA CCS '23"},"container-title":["Proceedings of the ACM Asia Conference on Computer and Communications Security"],"original-title":[],"deposited":{"date-parts":[[2023,7,5]],"date-time":"2023-07-05T15:10:50Z","timestamp":1688569850000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3579856.3582830"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,10]]},"references-count":29,"alternative-id":["10.1145\/3579856.3582830","10.1145\/3579856"],"URL":"https:\/\/doi.org\/10.1145\/3579856.3582830","relation":{},"subject":[],"published":{"date-parts":[[2023,7,10]]},"assertion":[{"value":"2023-07-10","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}