{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,11,19]],"date-time":"2024-11-19T18:31:04Z","timestamp":1732041064779},"reference-count":121,"publisher":"Association for Computing Machinery (ACM)","issue":"6","funder":[{"name":"Fonds National de la Recherche","award":["PRIDE15\/10621687"]},{"DOI":"10.13039\/100000308","name":"Carnegie Corporation of New York","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100000308","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput.-Hum. Interact."],"published-print":{"date-parts":[[2021,12,31]]},"abstract":"Usable privacy and security researchers have developed a variety of approaches to represent risk to research participants. To understand how these approaches are used and when each might be most appropriate, we conducted a systematic literature review of methods used in security and privacy studies with human participants. From a sample of 633 papers published at five top conferences between 2014 and 2018 that included keywords related to both security\/privacy and usability, we systematically selected and analyzed 284 full-length papers that included human subjects studies. Our analysis focused on study methods; risk representation; the use of prototypes, scenarios, and educational intervention; the use of deception to simulate risk; and types of participants. We discuss benefits and shortcomings of the methods, and identify key methodological, ethical, and research challenges when representing and assessing security and privacy risk. We also provide guidelines for the reporting of user studies in security and privacy.<\/jats:p>","DOI":"10.1145\/3469845","type":"journal-article","created":{"date-parts":[[2021,12,23]],"date-time":"2021-12-23T13:21:43Z","timestamp":1640265703000},"page":"1-50","update-policy":"http:\/\/dx.doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":26,"title":["A Systematic Literature Review of Empirical Methods and Risk Representation in Usable Privacy and Security Research"],"prefix":"10.1145","volume":"28","author":[{"given":"Verena","family":"Distler","sequence":"first","affiliation":[{"name":"University of Luxembourg"}]},{"given":"Matthias","family":"Fassl","sequence":"additional","affiliation":[{"name":"CISPA Helmholtz Center for Information Security, Esch-sur-Alzette Luxembourg"}]},{"given":"Hana","family":"Habib","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, United States"}]},{"given":"Katharina","family":"Krombholz","sequence":"additional","affiliation":[{"name":"CISPA Helmholtz Center for Information Security, Esch-sur-Alzette Luxembourg"}]},{"given":"Gabriele","family":"Lenzini","sequence":"additional","affiliation":[{"name":"University of Luxembourg"}]},{"given":"Carine","family":"Lallemand","sequence":"additional","affiliation":[{"name":"Eindhoven University of Technology & University of Luxembourg"}]},{"given":"Lorrie Faith","family":"Cranor","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, United States"}]},{"given":"Vincent","family":"Koenig","sequence":"additional","affiliation":[{"name":"University of Luxembourg"}]}],"member":"320","published-online":{"date-parts":[[2021,12,23]]},"reference":[{"key":"e_1_3_3_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2017.52"},{"key":"e_1_3_3_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3054926"},{"key":"e_1_3_3_4_1","doi-asserted-by":"publisher","DOI":"10.5555\/3291228.3291263"},{"key":"e_1_3_3_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2466246"},{"key":"e_1_3_3_6_1","doi-asserted-by":"publisher","DOI":"10.5555\/3235895.3235914"},{"key":"e_1_3_3_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702334"},{"key":"e_1_3_3_8_1","doi-asserted-by":"publisher","DOI":"10.5555\/3235924.3235929"},{"key":"e_1_3_3_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2639189.2641208"},{"key":"e_1_3_3_10_1","unstructured":"American Psychological Association. 2017. Ethical principles of psychologists and code of conduct (2002 Amended Effective June 1 2010 and January 1 2017). http:\/\/www.apa.org\/ethics\/code\/index.html."},{"key":"e_1_3_3_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702322"},{"key":"e_1_3_3_12_1","doi-asserted-by":"publisher","DOI":"10.5555\/3235866.3235869"},{"key":"e_1_3_3_13_1","doi-asserted-by":"publisher","DOI":"10.1258\/ce.2008.008047"},{"key":"e_1_3_3_14_1","doi-asserted-by":"publisher","DOI":"10.1037\/0003-066X.40.2.165"},{"key":"e_1_3_3_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2333112.2333114"},{"key":"e_1_3_3_16_1","doi-asserted-by":"publisher","DOI":"10.5555\/3235924.3235940"},{"key":"e_1_3_3_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2012.44"},{"key":"e_1_3_3_18_1","doi-asserted-by":"publisher","DOI":"10.5555\/3235838.3235847"},{"key":"e_1_3_3_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858498"},{"key":"e_1_3_3_20_1","doi-asserted-by":"publisher","DOI":"10.5555\/3235924.3235947"},{"key":"e_1_3_3_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2016.53"},{"key":"e_1_3_3_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2014.109"},{"key":"e_1_3_3_23_1","doi-asserted-by":"publisher","DOI":"10.5555\/3235924.3235938"},{"key":"e_1_3_3_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025991"},{"key":"e_1_3_3_25_1","unstructured":"Deception Research\u2014APA Dictionary of Psychology. (n.d.). Retrieved January 20 2020 from https:\/\/dictionary.apa.org\/deception-research."},{"key":"e_1_3_3_26_1","doi-asserted-by":"publisher","DOI":"10.5555\/3241094.3241110"},{"key":"e_1_3_3_27_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2019.106227"},{"key":"e_1_3_3_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702251"},{"key":"e_1_3_3_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1240866.1241089"},{"key":"e_1_3_3_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702249"},{"key":"e_1_3_3_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025636"},{"key":"e_1_3_3_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2501604.2501617"},{"key":"e_1_3_3_33_1","doi-asserted-by":"publisher","DOI":"10.5555\/3488905.3488912"},{"key":"e_1_3_3_34_1","doi-asserted-by":"publisher","DOI":"10.5555\/3241189.3241292"},{"key":"e_1_3_3_35_1","doi-asserted-by":"publisher","DOI":"10.5555\/3235895.3235897"},{"key":"e_1_3_3_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173627"},{"key":"e_1_3_3_37_1","first-page":"11","volume-title":"Security Behavior Observatory: Infrastructure for Long-Term Monitoring of Client Machines","author":"Forget A.","year":"2014","unstructured":"A. Forget, S. Komanduri, A. Acquisti, N. Christin, L. F. Cranor, and R. Telang. 2014. Security Behavior Observatory: Infrastructure for Long-Term Monitoring of Client Machines. Technical Report CMU-CyLab-14-009. CyLab, Carnegie Mellon University, p. 11."},{"key":"e_1_3_3_38_1","doi-asserted-by":"publisher","DOI":"10.5555\/3235895.3235904"},{"key":"e_1_3_3_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174241"},{"key":"e_1_3_3_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2017.10"},{"key":"e_1_3_3_41_1","doi-asserted-by":"publisher","DOI":"10.5555\/2755205"},{"key":"e_1_3_3_42_1","doi-asserted-by":"publisher","DOI":"10.5555\/3291228.3291242"},{"key":"e_1_3_3_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2976749.2978330"},{"key":"e_1_3_3_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174144"},{"key":"e_1_3_3_45_1","doi-asserted-by":"publisher","DOI":"10.5555\/3235866.3235881"},{"key":"e_1_3_3_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3369457.3369460"},{"key":"e_1_3_3_47_1","doi-asserted-by":"publisher","DOI":"10.5555\/3235838.3235859"},{"key":"e_1_3_3_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2556978"},{"key":"e_1_3_3_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858139"},{"key":"e_1_3_3_50_1","doi-asserted-by":"publisher","DOI":"10.5555\/3277203.3277285"},{"key":"e_1_3_3_51_1","doi-asserted-by":"publisher","DOI":"10.5555\/3235866.3235883"},{"key":"e_1_3_3_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025788"},{"key":"e_1_3_3_53_1","doi-asserted-by":"publisher","DOI":"10.1561\/1100000004"},{"key":"e_1_3_3_54_1","doi-asserted-by":"publisher","DOI":"10.5555\/3235838.3235865"},{"key":"e_1_3_3_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/1290958.1290968"},{"key":"e_1_3_3_56_1","doi-asserted-by":"publisher","DOI":"10.5555\/3291228.3291246"},{"key":"e_1_3_3_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357127"},{"key":"e_1_3_3_58_1","doi-asserted-by":"publisher","DOI":"10.5555\/2671225.2671263"},{"key":"e_1_3_3_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/1572532.1572536"},{"key":"e_1_3_3_60_1","doi-asserted-by":"publisher","DOI":"10.5555\/3235924.3235943"},{"key":"e_1_3_3_61_1","doi-asserted-by":"publisher","DOI":"10.5555\/3488905.3488909"},{"key":"e_1_3_3_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025879"},{"key":"e_1_3_3_63_1","doi-asserted-by":"publisher","DOI":"10.5555\/3277203.3277220"},{"key":"e_1_3_3_64_1","doi-asserted-by":"publisher","DOI":"10.5555\/3235895.3235912"},{"key":"e_1_3_3_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858085"},{"key":"e_1_3_3_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025735"},{"key":"e_1_3_3_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858384"},{"key":"e_1_3_3_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702127"},{"key":"e_1_3_3_69_1","doi-asserted-by":"publisher","DOI":"10.5555\/3235895.3235920"},{"key":"e_1_3_3_70_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.309.6954.597"},{"key":"e_1_3_3_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/3133956.3134082"},{"key":"e_1_3_3_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/3133956.3134082"},{"key":"e_1_3_3_73_1","doi-asserted-by":"publisher","DOI":"10.5555\/3291228.3291252"},{"key":"e_1_3_3_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/1520340.1520401"},{"key":"e_1_3_3_75_1","doi-asserted-by":"publisher","DOI":"10.5555\/2831143.2831159"},{"key":"e_1_3_3_76_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.im.2014.08.008"},{"key":"e_1_3_3_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025482"},{"key":"e_1_3_3_78_1","doi-asserted-by":"publisher","DOI":"10.5555\/3241189.3241220"},{"key":"e_1_3_3_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174035"},{"key":"e_1_3_3_80_1","doi-asserted-by":"publisher","DOI":"10.5555\/3291228.3291232"},{"key":"e_1_3_3_81_1","volume-title":"Privacy and Information Sharing","author":"Rainie L.","year":"2015","unstructured":"L. Rainie and M. Duggan. 2015. Privacy and Information Sharing. Pew Research Center."},{"key":"e_1_3_3_82_1","doi-asserted-by":"publisher","DOI":"10.5555\/3235838.3235864"},{"key":"e_1_3_3_83_1","doi-asserted-by":"publisher","DOI":"10.1002\/bsl.906"},{"key":"e_1_3_3_84_1","doi-asserted-by":"publisher","DOI":"10.1145\/2976749.2978307"},{"key":"e_1_3_3_85_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219166.3219185"},{"key":"e_1_3_3_86_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174086"},{"key":"e_1_3_3_87_1","doi-asserted-by":"publisher","DOI":"10.1145\/2660267.2660305"},{"key":"e_1_3_3_88_1","doi-asserted-by":"publisher","DOI":"10.1375\/pplt.2003.10.2.289"},{"key":"e_1_3_3_89_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858400"},{"key":"e_1_3_3_90_1","doi-asserted-by":"publisher","DOI":"10.5555\/3291228.3291258"},{"key":"e_1_3_3_91_1","doi-asserted-by":"publisher","DOI":"10.5555\/3235924.3235950"},{"key":"e_1_3_3_92_1","doi-asserted-by":"publisher","DOI":"10.5555\/3235924.3235954"},{"key":"e_1_3_3_93_1","doi-asserted-by":"publisher","DOI":"10.5555\/3291228.3291240"},{"key":"e_1_3_3_94_1","doi-asserted-by":"publisher","DOI":"10.1145\/3170427.3188511"},{"key":"e_1_3_3_95_1","volume-title":"Common Pitfalls in Writing about Security and Privacy Human Subjects Experiments, and How to Avoid ThemMSR-TR-2013-5. Microsoft Technical Report","author":"Schechter S.","year":"2013","unstructured":"S. Schechter. 2013a. Common Pitfalls in Writing about Security and Privacy Human Subjects Experiments, and How to Avoid Them. MSR-TR-2013-5. Microsoft Technical Report. https:\/\/www.microsoft.com\/en-us\/research\/publication\/common-pitfalls-in-writing-about-security-and-privacy-human-subjects-experiments-and-how-to-avoid-them\/."},{"key":"e_1_3_3_96_1","volume-title":"Proceedings of the Workshop on Home Usable Privacy and Security (HUPS\u201913)","author":"Schechter S.","year":"2013","unstructured":"S. Schechter. 2013b. The user is the enemy, and (S)he keeps reaching for that bright shiny power button! In Proceedings of the Workshop on Home Usable Privacy and Security (HUPS\u201913). https:\/\/www.microsoft.com\/en-us\/research\/publication\/the-user-is-the-enemy-and-she-keeps-reaching-for-that-bright-shiny-power-button\/."},{"key":"e_1_3_3_97_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2007.35"},{"key":"e_1_3_3_98_1","doi-asserted-by":"publisher","DOI":"10.5555\/3235924.3235926"},{"key":"e_1_3_3_99_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2014.28"},{"key":"e_1_3_3_100_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557330"},{"key":"e_1_3_3_101_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702586"},{"key":"e_1_3_3_102_1","doi-asserted-by":"publisher","DOI":"10.1145\/2660267.2660274"},{"key":"e_1_3_3_103_1","doi-asserted-by":"publisher","DOI":"10.1145\/2976749.2978328"},{"key":"e_1_3_3_104_1","doi-asserted-by":"publisher","DOI":"10.1145\/2078827.2078831"},{"key":"e_1_3_3_105_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025851"},{"key":"e_1_3_3_106_1","volume-title":"Introduction to Psychology","author":"Stangor C.","year":"2018","unstructured":"C. Stangor and J. Walinga. 2018. Introduction to Psychology (1st Canadian ed.). BCcampus Open Publishing. https:\/\/opentextbc.ca\/introductiontopsychology\/.","edition":"1"},{"key":"e_1_3_3_107_1","doi-asserted-by":"publisher","DOI":"10.5555\/3277203.3277250"},{"key":"e_1_3_3_108_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025733"},{"key":"e_1_3_3_109_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3026050"},{"key":"e_1_3_3_110_1","doi-asserted-by":"publisher","DOI":"10.5555\/3235866.3235877"},{"key":"e_1_3_3_111_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025901"},{"key":"e_1_3_3_112_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557275"},{"key":"e_1_3_3_113_1","doi-asserted-by":"publisher","DOI":"10.5555\/3291228.3291233"},{"key":"e_1_3_3_114_1","doi-asserted-by":"publisher","DOI":"10.5555\/3235924.3235928"},{"key":"e_1_3_3_115_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2017.09.012"},{"key":"e_1_3_3_116_1","doi-asserted-by":"publisher","DOI":"10.5555\/3291228.3291230"},{"key":"e_1_3_3_117_1","doi-asserted-by":"publisher","DOI":"10.5555\/3291228.3291247"},{"key":"e_1_3_3_118_1","doi-asserted-by":"publisher","DOI":"10.5555\/3235895.3235918"},{"key":"e_1_3_3_119_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174066"},{"key":"e_1_3_3_120_1","doi-asserted-by":"publisher","DOI":"10.5555\/3235838.3235846"},{"key":"e_1_3_3_121_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858270"},{"key":"e_1_3_3_122_1","doi-asserted-by":"publisher","DOI":"10.5555\/3291228.3291245"}],"container-title":["ACM Transactions on Computer-Human Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3469845","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,2]],"date-time":"2023-01-02T01:43:01Z","timestamp":1672623781000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3469845"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,23]]},"references-count":121,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2021,12,31]]}},"alternative-id":["10.1145\/3469845"],"URL":"https:\/\/doi.org\/10.1145\/3469845","relation":{},"ISSN":["1073-0516","1557-7325"],"issn-type":[{"value":"1073-0516","type":"print"},{"value":"1557-7325","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,12,23]]},"assertion":[{"value":"2020-07-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-06-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-12-23","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}