{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,9,7]],"date-time":"2024-09-07T22:57:37Z","timestamp":1725749857743},"reference-count":60,"publisher":"Association for Computing Machinery (ACM)","issue":"3-4","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Interact. Intell. Syst."],"published-print":{"date-parts":[[2021,12,31]]},"abstract":"The adoption of artificial intelligence (AI) systems in environments that involve high risk and high consequence decision-making is severely hampered by critical design issues. These issues include system transparency and brittleness, where transparency relates to (i) the explainability of results and (ii) the ability of a user to inspect and verify system goals and constraints; and brittleness, (iii) the ability of a system to adapt to new user demands. Transparency is a particular concern for criminal intelligence analysis, where there are significant ethical and trust issues that arise when algorithmic and system processes are not adequately understood by a user. This prevents adoption of potentially useful technologies in policing environments. In this article, we present a novel approach to designing a conversational agent (CA) AI system for intelligence analysis that tackles these issues. We discuss the results and implications of three different studies; a Cognitive Task Analysis to understand analyst thinking when retrieving information in an investigation, Emergent Themes Analysis to understand the explanation needs of different system components, and an interactive experiment with a prototype conversational agent. Our prototype conversational agent, named Pan, demonstrates transparency provision and mitigates brittleness by evolving new CA intentions. We encode interactions with the CA with human factors principles for situation recognition and use interactive visual analytics to support analyst reasoning. Our approach enables complex AI systems, such as Pan, to be used in sensitive environments, and our research has broader application than the use case discussed.<\/jats:p>","DOI":"10.1145\/3444369","type":"journal-article","created":{"date-parts":[[2021,9,3]],"date-time":"2021-09-03T19:30:07Z","timestamp":1630697407000},"page":"1-35","update-policy":"http:\/\/dx.doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Developing Conversational Agents for Use in Criminal Investigations"],"prefix":"10.1145","volume":"11","author":[{"given":"Sam","family":"Hepenstal","sequence":"first","affiliation":[{"name":"Defence Science Technology Laboratory, Porton Down, Salisbury, UK"}]},{"given":"Leishi","family":"Zhang","sequence":"additional","affiliation":[{"name":"Middlesex University London, The Burroughs, London, UK"}]},{"given":"Neesha","family":"Kodagoda","sequence":"additional","affiliation":[{"name":"Middlesex University London, The Burroughs, London, UK"}]},{"given":"B. l. william","family":"Wong","sequence":"additional","affiliation":[{"name":"Middlesex University London, The Burroughs, London, UK"}]}],"member":"320","published-online":{"date-parts":[[2021,9,3]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"DeepPavlov: An Open Source Conversational AI Framework. Retrieved on 12th","year":"2021","unstructured":"[n.d.]. DeepPavlov: An Open Source Conversational AI Framework. Retrieved on 12th Jan 2021 from http:\/\/deeppavlov.ai\/. [n.d.]. DeepPavlov: An Open Source Conversational AI Framework. Retrieved on 12th Jan 2021 from http:\/\/deeppavlov.ai\/."},{"key":"e_1_2_1_2_1","volume-title":"Retrieved on 12th","author":"Language Understanding","year":"2021","unstructured":"[n.d.]. Language Understanding (LUIS). Retrieved on 12th Jan 2021 from https:\/\/www.luis.ai\/home. [n.d.]. Language Understanding (LUIS). Retrieved on 12th Jan 2021 from https:\/\/www.luis.ai\/home."},{"key":"e_1_2_1_3_1","volume-title":"Learn AI-designing and Architecting Intelligent Agents. Retrieved on 12th","year":"2021","unstructured":"[n.d.]. Learn AI-designing and Architecting Intelligent Agents. Retrieved on 12th Jan 2021 from https:\/\/azure.github.io\/LearnAI-DesigningandArchitectingIntelligentAgents\/. [n.d.]. Learn AI-designing and Architecting Intelligent Agents. Retrieved on 12th Jan 2021 from https:\/\/azure.github.io\/LearnAI-DesigningandArchitectingIntelligentAgents\/."},{"key":"#cr-split#-e_1_2_1_4_1.1","doi-asserted-by":"crossref","unstructured":"Simon Andrews Babak Akhgar Simeon Yates Alex Stedmon and Laurence Hirsch. 2014. Using formal concept analysis to detect and monitor organised crime. Lect. Notes Comput. Sci. 8132. DOI:https:\/\/doi.org\/10.1007\/978-3-642-40769-7_11 10.1007\/978-3-642-40769-7_11","DOI":"10.1007\/978-3-642-40769-7_11"},{"key":"#cr-split#-e_1_2_1_4_1.2","doi-asserted-by":"crossref","unstructured":"Simon Andrews Babak Akhgar Simeon Yates Alex Stedmon and Laurence Hirsch. 2014. Using formal concept analysis to detect and monitor organised crime. Lect. Notes Comput. Sci. 8132. DOI:https:\/\/doi.org\/10.1007\/978-3-642-40769-7_11","DOI":"10.1007\/978-3-642-40769-7_11"},{"key":"e_1_2_1_5_1","volume-title":"Facets of Systems Science","author":"Ashby W. Ross","unstructured":"W. Ross Ashby . 1991. Requisite variety and its implications for the control of complex systems . In Facets of Systems Science . Springer US , Boston, MA , 405\u2013417. DOI:https:\/\/doi.org\/10.1007\/978-1-4899-0718-9_28 10.1007\/978-1-4899-0718-9_28 W. Ross Ashby. 1991. Requisite variety and its implications for the control of complex systems. In Facets of Systems Science. Springer US, Boston, MA, 405\u2013417. DOI:https:\/\/doi.org\/10.1007\/978-1-4899-0718-9_28"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.5465\/amr.1998.926621"},{"key":"e_1_2_1_7_1","first-page":"421","article-title":"Situation awareness in emergency medical dispatch. Int. J. Hum.-comput","volume":"61","author":"Blandford Ann","year":"2004","unstructured":"Ann Blandford and B. L. William Wong . 2004 . Situation awareness in emergency medical dispatch. Int. J. Hum.-comput . Stud. 61 , 4 (2004), 421 \u2013 452 . DOI:https:\/\/doi.org\/10.1016\/j.ijhcs.2003.12.012 10.1016\/j.ijhcs.2003.12.012 Ann Blandford and B. L. William Wong. 2004. Situation awareness in emergency medical dispatch. Int. J. Hum.-comput. Stud. 61, 4 (2004), 421\u2013452. DOI:https:\/\/doi.org\/10.1016\/j.ijhcs.2003.12.012","journal-title":"Stud."},{"key":"e_1_2_1_8_1","volume-title":"Moran","author":"Card Stuart K.","year":"1983","unstructured":"Stuart K. Card , Allen Newell , and Thomas P . Moran . 1983 . The Psychology of Human-computer Interaction. L. Erlbaum Associates Inc ., Hillsdale, NJ. Stuart K. Card, Allen Newell, and Thomas P. Moran. 1983. The Psychology of Human-computer Interaction. L. Erlbaum Associates Inc., Hillsdale, NJ."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1070"},{"key":"e_1_2_1_10_1","unstructured":"Hannah Couchman. 2019. Policing by machine: Predictive policing and the threat to our rights. Liberty. https:\/\/www.libertyhumanrights.org.uk\/issue\/policing-by-machine\/. Hannah Couchman. 2019. Policing by machine: Predictive policing and the threat to our rights. Liberty. https:\/\/www.libertyhumanrights.org.uk\/issue\/policing-by-machine\/."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1177\/1071181319631264"},{"key":"e_1_2_1_12_1","volume-title":"Formal Concept Analysis","author":"Garriga Gemma C.","unstructured":"Gemma C. Garriga . 2017. Formal Concept Analysis . Springer US , Boston, MA , 522\u2013523. DOI:https:\/\/doi.org\/10.1007\/978-1-4899-7687-1_316 10.1007\/978-1-4899-7687-1_316 Gemma C. Garriga. 2017. Formal Concept Analysis. Springer US, Boston, MA, 522\u2013523. DOI:https:\/\/doi.org\/10.1007\/978-1-4899-7687-1_316"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1177\/1541931213601039"},{"key":"e_1_2_1_14_1","volume-title":"Explaining Explanations: An Approach to Evaluating Interpretability of Machine Learning.","author":"Gilpin Leilani H.","year":"2018","unstructured":"Leilani H. Gilpin , David Bau , Ben Z. Yuan , Ayesha Bajwa , Michael Specter , and Lalana Kagal . 2018 . Explaining Explanations: An Approach to Evaluating Interpretability of Machine Learning. Retrieved from http:\/\/arxiv.org\/abs\/1806.00069. Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. 2018. Explaining Explanations: An Approach to Evaluating Interpretability of Machine Learning. Retrieved from http:\/\/arxiv.org\/abs\/1806.00069."},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the Workshop on Algorithmic Transparency in Emerging Technologies co-located with 24th International Conference on Intelligent User Interfaces\u201919)","author":"Hepenstal Sam","unstructured":"Sam Hepenstal , Neesha Kodagoda , Leishi Zhang , Pragya Paudyal , and B. L. William Wong . 2019. Algorithmic transparency of conversational agents . In Proceedings of the Workshop on Algorithmic Transparency in Emerging Technologies co-located with 24th International Conference on Intelligent User Interfaces\u201919) . Sam Hepenstal, Neesha Kodagoda, Leishi Zhang, Pragya Paudyal, and B. L. William Wong. 2019. Algorithmic transparency of conversational agents. In Proceedings of the Workshop on Algorithmic Transparency in Emerging Technologies co-located with 24th International Conference on Intelligent User Interfaces\u201919)."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1177\/1071181319631333"},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of the 25th International Conference on Intelligent User Interfaces. ACM, 134\u2013135","author":"Hepenstal Sam","unstructured":"Sam Hepenstal , Leishi Zhang , Neesha Kodagoda , and B. L. William Wong . 2020. Pan: Conversational agent for criminal investigations . In Proceedings of the 25th International Conference on Intelligent User Interfaces. ACM, 134\u2013135 . DOI:https:\/\/doi.org\/10.1145\/3379336.3381463 10.1145\/3379336.3381463 Sam Hepenstal, Leishi Zhang, Neesha Kodagoda, and B. L. William Wong. 2020. Pan: Conversational agent for criminal investigations. In Proceedings of the 25th International Conference on Intelligent User Interfaces. ACM, 134\u2013135. DOI:https:\/\/doi.org\/10.1145\/3379336.3381463"},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the 64th Human Factors and Ergonomics Society Meeting.","author":"Hepenstal Sam","unstructured":"Sam Hepenstal , Leishi Zhang , Neesha Kodagoda , and B. L. William Wong . 2020. Providing a foundation for interpretable autonomous agents through elicitation and modelling of criminal investigation pathways . In Proceedings of the 64th Human Factors and Ergonomics Society Meeting. Sam Hepenstal, Leishi Zhang, Neesha Kodagoda, and B. L. William Wong. 2020. Providing a foundation for interpretable autonomous agents through elicitation and modelling of criminal investigation pathways. In Proceedings of the 64th Human Factors and Ergonomics Society Meeting."},{"key":"e_1_2_1_19_1","unstructured":"Sam Hepenstal Leishi Zhang Neesha Kodagoda and B. L. William Wong. 2020. What are you Thinking? Explaining conversation agent responses for criminal investigations. In Proceedings of the Workshop on Explainable Smart Systems for Algorithmic Transparency in Emerging Technologies co-located with 25th International Conference on Intelligent User Interfaces (IUI\u201920) Cagliari Italy March 17 2020 (CEUR Workshop Proceedings) Alison Smith-Renner Styliani Kleanthous Brian Lim Tsvi Kuflik Simone Stumpf Jahna Otterbacher Advait Sarkar Casey Dugan and Avital Shulner Tal (Eds.) Vol. 2582. CEUR-WS.org. Retrieved from http:\/\/ceur-ws.org\/Vol-2582\/paper3.pdf. Sam Hepenstal Leishi Zhang Neesha Kodagoda and B. L. William Wong. 2020. What are you Thinking? Explaining conversation agent responses for criminal investigations. In Proceedings of the Workshop on Explainable Smart Systems for Algorithmic Transparency in Emerging Technologies co-located with 25th International Conference on Intelligent User Interfaces (IUI\u201920) Cagliari Italy March 17 2020 (CEUR Workshop Proceedings) Alison Smith-Renner Styliani Kleanthous Brian Lim Tsvi Kuflik Simone Stumpf Jahna Otterbacher Advait Sarkar Casey Dugan and Avital Shulner Tal (Eds.) Vol. 2582. CEUR-WS.org. Retrieved from http:\/\/ceur-ws.org\/Vol-2582\/paper3.pdf."},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the Human Factors and Ergonomics Society Meeting 62","author":"Hoffman Robert R.","year":"2018","unstructured":"Robert R. Hoffman , Gary Klein , and Shane T. Mueller . 2018. Explaining explanation For \u201cExplainable AI \u201d. Proceedings of the Human Factors and Ergonomics Society Meeting 62 , 1 ( 2018 ), 197\u2013201. DOI:https:\/\/doi.org\/10.1177\/1541931218621047 10.1177\/1541931218621047 Robert R. Hoffman, Gary Klein, and Shane T. Mueller. 2018. Explaining explanation For \u201cExplainable AI\u201d. Proceedings of the Human Factors and Ergonomics Society Meeting 62, 1 (2018), 197\u2013201. DOI:https:\/\/doi.org\/10.1177\/1541931218621047"},{"key":"e_1_2_1_21_1","article-title":"Machine learning methods and applications using formal concept analysis","volume":"2","author":"Jabin Suraiya","year":"2015","unstructured":"Suraiya Jabin . 2015 . Machine learning methods and applications using formal concept analysis . Int. J. New Technol. Sci. Eng. 2 , 3 (2015). Suraiya Jabin. 2015. Machine learning methods and applications using formal concept analysis. Int. J. New Technol. Sci. Eng. 2, 3 (2015).","journal-title":"Int. J. New Technol. Sci. Eng."},{"key":"e_1_2_1_22_1","unstructured":"Bret Kinsella. 2018. Amazon Echo Device Sales Break New Records Alexa Tops Free App Downloads for iOS and Android and Alexa Down in Europe on Christmas Morning. Retrieved from https:\/\/voicebot.ai\/2018\/12\/26\/amazon-echo-device-sales-break-new-records-alexa-tops-free-app-downloads-for-ios-and-android-and-alexa-down-in-europe-on-christmas-morning\/. Bret Kinsella. 2018. Amazon Echo Device Sales Break New Records Alexa Tops Free App Downloads for iOS and Android and Alexa Down in Europe on Christmas Morning. Retrieved from https:\/\/voicebot.ai\/2018\/12\/26\/amazon-echo-device-sales-break-new-records-alexa-tops-free-app-downloads-for-ios-and-android-and-alexa-down-in-europe-on-christmas-morning\/."},{"key":"e_1_2_1_23_1","unstructured":"Bret Kinsella. 2019. NPR Study Says 118 Million Smart Speakers Owned by U.S. Adults. Retrieved from https:\/\/voicebot.ai\/2019\/01\/07\/npr-study-says-118-million-smart-speakers-owned-by-u-s-adults\/. Bret Kinsella. 2019. NPR Study Says 118 Million Smart Speakers Owned by U.S. Adults. Retrieved from https:\/\/voicebot.ai\/2019\/01\/07\/npr-study-says-118-million-smart-speakers-owned-by-u-s-adults\/."},{"key":"e_1_2_1_24_1","volume-title":"A Recognition .rimed Decision (RPD) model of rapid decision making. Decision Mak. Action: Model. Meth. (01","author":"Klein Gary","year":"1993","unstructured":"Gary Klein . 1993. A Recognition .rimed Decision (RPD) model of rapid decision making. Decision Mak. Action: Model. Meth. (01 1993 ). Gary Klein. 1993. A Recognition .rimed Decision (RPD) model of rapid decision making. Decision Mak. Action: Model. Meth. (01 1993)."},{"key":"e_1_2_1_25_1","volume-title":"Seeing What Others Don\u2019t","author":"Klein Gary","unstructured":"Gary Klein . 2017. Seeing What Others Don\u2019t . Nicholas Brearley Publishing . Gary Klein. 2017. Seeing What Others Don\u2019t. Nicholas Brearley Publishing."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/21.31053"},{"key":"e_1_2_1_27_1","volume-title":"Making sense of sensemaking 2: A macrocognitive model. Intell. Syst. 21 (10","author":"Klein Gary","year":"2006","unstructured":"Gary Klein , Brian Moon , and Robert Hoffman . 2006. Making sense of sensemaking 2: A macrocognitive model. Intell. Syst. 21 (10 2006 ), 88\u201392. DOI:https:\/\/doi.org\/10.1109\/MIS.2006.100 10.1109\/MIS.2006.100 Gary Klein, Brian Moon, and Robert Hoffman. 2006. Making sense of sensemaking 2: A macrocognitive model. Intell. Syst. 21 (10 2006), 88\u201392. DOI:https:\/\/doi.org\/10.1109\/MIS.2006.100"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1037\/e578382012-033"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1168149.1168168"},{"key":"e_1_2_1_30_1","volume-title":"Understanding artificial intelligence ethics and safety. CoRR abs\/1906.05684","author":"Leslie David","year":"2019","unstructured":"David Leslie . 2019. Understanding artificial intelligence ethics and safety. CoRR abs\/1906.05684 ( 2019 ). David Leslie. 2019. Understanding artificial intelligence ethics and safety. CoRR abs\/1906.05684 (2019)."},{"key":"#cr-split#-e_1_2_1_31_1.1","doi-asserted-by":"crossref","unstructured":"Georgios Leventakis and M. R. Haberfeld. 2018. Societal Implications of Community-oriented Policing and Technology. Springer. DOI:https:\/\/doi.org\/10.1007\/978-3-319-89297-9 10.1007\/978-3-319-89297-9","DOI":"10.1007\/978-3-319-89297-9"},{"key":"#cr-split#-e_1_2_1_31_1.2","doi-asserted-by":"crossref","unstructured":"Georgios Leventakis and M. R. Haberfeld. 2018. Societal Implications of Community-oriented Policing and Technology. Springer. DOI:https:\/\/doi.org\/10.1007\/978-3-319-89297-9","DOI":"10.1007\/978-3-319-89297-9"},{"key":"e_1_2_1_32_1","volume-title":"The mythos of model interpretability. CoRR abs\/1606.03490","author":"Lipton Zachary Chase","year":"2016","unstructured":"Zachary Chase Lipton . 2016. The mythos of model interpretability. CoRR abs\/1606.03490 ( 2016 ). Zachary Chase Lipton. 2016. The mythos of model interpretability. CoRR abs\/1606.03490 (2016)."},{"key":"e_1_2_1_33_1","unstructured":"Bernard Marr. 2014. Dear IKEA: Your customer service is terrible. LinkedIn. Retrieved from www.linkedin.com\/pulse\/20140325060328-64875646-dear-ikea-your-customer-service-is-terrible. Bernard Marr. 2014. Dear IKEA: Your customer service is terrible. LinkedIn. Retrieved from www.linkedin.com\/pulse\/20140325060328-64875646-dear-ikea-your-customer-service-is-terrible."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/505282.505285"},{"key":"e_1_2_1_35_1","volume-title":"Proceedings of the International Conference on Language Resources and Evaluation (LREC\u201918)","author":"Mikolov Tomas","year":"2018","unstructured":"Tomas Mikolov , Edouard Grave , Piotr Bojanowski , Christian Puhrsch , and Armand Joulin . 2018 . Advances in pre-training distributed word representations . In Proceedings of the International Conference on Language Resources and Evaluation (LREC\u201918) . Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the International Conference on Language Resources and Evaluation (LREC\u201918)."},{"key":"e_1_2_1_36_1","volume-title":"Interpretable Machine Learning. A Guide for Making Black Box Models Explainable","author":"Molnar Christoph","unstructured":"Christoph Molnar . 2019. Interpretable Machine Learning. A Guide for Making Black Box Models Explainable ( 1 st ed.). Lulu . Christoph Molnar. 2019. Interpretable Machine Learning. A Guide for Making Black Box Models Explainable (1st ed.). Lulu.","edition":"1"},{"key":"e_1_2_1_37_1","volume-title":"Design rules based on analyses of human error.Commun. ACM 26 (04","author":"Norman Donald","year":"1983","unstructured":"Donald Norman . 1983. Design rules based on analyses of human error.Commun. ACM 26 (04 1983 ), 254\u2013258. DOI:https:\/\/doi.org\/10.1145\/2163.358092 10.1145\/2163.358092 Donald Norman. 1983. Design rules based on analyses of human error.Commun. ACM 26 (04 1983), 254\u2013258. DOI:https:\/\/doi.org\/10.1145\/2163.358092"},{"key":"e_1_2_1_38_1","volume-title":"Explaining the evidence: Tests of the story model for juror decision making. J. Personal. Soc. Psychol.ogy 62 (02","author":"Pennington Nancy","year":"1992","unstructured":"Nancy Pennington and Reid Hastie . 1992. Explaining the evidence: Tests of the story model for juror decision making. J. Personal. Soc. Psychol.ogy 62 (02 1992 ), 189\u2013206. DOI:https:\/\/doi.org\/10.1037\/0022-3514.62.2.189 10.1037\/0022-3514.62.2.189 Nancy Pennington and Reid Hastie. 1992. Explaining the evidence: Tests of the story model for juror decision making. J. Personal. Soc. Psychol.ogy 62 (02 1992), 189\u2013206. DOI:https:\/\/doi.org\/10.1037\/0022-3514.62.2.189"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/THMS.2017.2700625"},{"key":"e_1_2_1_40_1","unstructured":"Eric Prud\u2019hommeaux and Andy Seaborne. 2007. SPARQL Query Language For RDF. Retrieved from https:\/\/www.w3.org\/TR\/2008\/REC-rdf-sparql-query-20080115\/. Eric Prud\u2019hommeaux and Andy Seaborne. 2007. SPARQL Query Language For RDF. Retrieved from https:\/\/www.w3.org\/TR\/2008\/REC-rdf-sparql-query-20080115\/."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/SMC.2016.7844519"},{"key":"e_1_2_1_42_1","volume-title":"Benton","author":"Radziwill Nicole M.","year":"2017","unstructured":"Nicole M. Radziwill and Morgan C . Benton . 2017 . Evaluating quality of chatbots and intelligent conversational agents. CoRR abs\/1704.04579 (2017). Nicole M. Radziwill and Morgan C. Benton. 2017. Evaluating quality of chatbots and intelligent conversational agents. CoRR abs\/1704.04579 (2017)."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2014.2346481"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1177\/0018720816634228"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2019.03.033"},{"key":"e_1_2_1_46_1","volume-title":"What makes a good conversation? How controllable attributes affect human judgments. CoRR abs\/1902.08654","author":"Roller Stephen","year":"2019","unstructured":"Abigail See, Stephen Roller , Douwe Kiela , and Jason Weston . 2019. What makes a good conversation? How controllable attributes affect human judgments. CoRR abs\/1902.08654 ( 2019 ). Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good conversation? How controllable attributes affect human judgments. CoRR abs\/1902.08654 (2019)."},{"key":"e_1_2_1_47_1","unstructured":"Danny Shaw. 2019. Crime solving rates \u201cwoefully low \u201d Met Police Commissioner says. BBC. Retrieved from https:\/\/www.bbc.co.uk\/news\/uk-48780585. Danny Shaw. 2019. Crime solving rates \u201cwoefully low \u201d Met Police Commissioner says. BBC. Retrieved from https:\/\/www.bbc.co.uk\/news\/uk-48780585."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3301275.3302322"},{"key":"e_1_2_1_49_1","volume-title":"Cook.2005. Illuminating the Path: The Research and Development Agenda for Visual Analytics. Pacific Northwest National Laboratory","author":"Thomas James J.","unstructured":"James J. Thomas and Kristin A . Cook.2005. Illuminating the Path: The Research and Development Agenda for Visual Analytics. Pacific Northwest National Laboratory , Richland, WA . James J. Thomas and Kristin A. Cook.2005. Illuminating the Path: The Research and Development Agenda for Visual Analytics. Pacific Northwest National Laboratory, Richland, WA."},{"key":"e_1_2_1_50_1","volume-title":"The Uses of Argument","author":"Toulmin Stephen E.","unstructured":"Stephen E. Toulmin . 1958. The Uses of Argument . Cambridge University Press . Stephen E. Toulmin. 1958. The Uses of Argument. Cambridge University Press."},{"key":"e_1_2_1_51_1","unstructured":"Jane Wakefield. 2016. Would you want to talk to a machine?BBC. Retrieved from https:\/\/www.bbc.co.uk\/news\/technology-36225980. Jane Wakefield. 2016. Would you want to talk to a machine?BBC. Retrieved from https:\/\/www.bbc.co.uk\/news\/technology-36225980."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3301275.3302317"},{"key":"e_1_2_1_53_1","volume-title":"Critical Decision Method Data Analysis","author":"William Wong B. L.","unstructured":"B. L. William Wong . 2003. Critical Decision Method Data Analysis . Lawrence Erlbaum Associates , 327\u2013346. B. L. William Wong. 2003. Critical Decision Method Data Analysis. Lawrence Erlbaum Associates, 327\u2013346."},{"key":"e_1_2_1_54_1","unstructured":"B. L. William Wong and Ann Blandford. 2002. Analysing ambulance dispatcher decision making: Trialing emergent themes analysis. In Proceedings of the HF2002 Human Factors Conference Design for the Whole Person - Integrating Physical Cognitive and Social Aspects: A Joint Conference of the Ergonomics Society of Australia (ESA) and the Computer Human Interaction Special Interest Group.. B. L. William Wong and Ann Blandford. 2002. Analysing ambulance dispatcher decision making: Trialing emergent themes analysis. In Proceedings of the HF2002 Human Factors Conference Design for the Whole Person - Integrating Physical Cognitive and Social Aspects: A Joint Conference of the Ergonomics Society of Australia (ESA) and the Computer Human Interaction Special Interest Group.."},{"key":"e_1_2_1_55_1","volume-title":"Proceedings of the Human Factors and Ergonomics Society Meeting 60 (09 2016","author":"William Wong B. L.","year":"2016","unstructured":"B. L. William Wong and Neesha Kodagoda . 2016 . How analysts think: Anchoring, laddering and associations . In Proceedings of the Human Factors and Ergonomics Society Meeting 60 (09 2016 ), 178\u2013182. DOI:https:\/\/doi.org\/10.1177\/1541931213601040 10.1177\/1541931213601040 B. L. William Wong and Neesha Kodagoda. 2016. How analysts think: Anchoring, laddering and associations. In Proceedings of the Human Factors and Ergonomics Society Meeting 60 (09 2016), 178\u2013182. DOI:https:\/\/doi.org\/10.1177\/1541931213601040"},{"key":"e_1_2_1_56_1","volume-title":"Proceedings of the 7th National Conference on Artificial Intelligence,. 127\u2013134","author":"Yevtushenko Serhiy A.","year":"2000","unstructured":"Serhiy A. Yevtushenko . 2000 . System of data analysis \u201cConcept Explorer.\u201d (In Russian) . In Proceedings of the 7th National Conference on Artificial Intelligence,. 127\u2013134 . Serhiy A. Yevtushenko. 2000. System of data analysis \u201cConcept Explorer.\u201d (In Russian). In Proceedings of the 7th National Conference on Artificial Intelligence,. 127\u2013134."},{"key":"e_1_2_1_57_1","unstructured":"Xiaoyu Yin Dagmar Gromann and Sebastian Rudolph. 2019. Neural Machine Translating from Natural Language to SPARQL. Retrieved from http:\/\/arxiv.org\/abs\/1906.09302. Xiaoyu Yin Dagmar Gromann and Sebastian Rudolph. 2019. Neural Machine Translating from Natural Language to SPARQL. Retrieved from http:\/\/arxiv.org\/abs\/1906.09302."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3301275.3308445"}],"container-title":["ACM Transactions on Interactive Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3444369","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,1]],"date-time":"2023-01-01T18:39:24Z","timestamp":1672598364000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3444369"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,3]]},"references-count":60,"journal-issue":{"issue":"3-4","published-print":{"date-parts":[[2021,12,31]]}},"alternative-id":["10.1145\/3444369"],"URL":"https:\/\/doi.org\/10.1145\/3444369","relation":{},"ISSN":["2160-6455","2160-6463"],"issn-type":[{"value":"2160-6455","type":"print"},{"value":"2160-6463","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,3]]},"assertion":[{"value":"2019-11-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-12-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-09-03","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}