Abstract
Human resource training, screening, and recruiting processes all require low-cost individual evaluation of participants in group discussions. This study aims to explore efficient ways to evaluate verbal discussions based on transcription data in an objective and multi-angle perspective to structure and annotate discussions for quantitative evaluation of the contribution from participants. Using two discussion data sets, this study conducted an experiment in which discussions were classified into a multi-layered discussion structure, and then annotations were further assigned within each layer and evaluated using a rating formula. In this experiment, we evaluated each participant in five evaluation dimensions and the result shows that they can be evaluated from multiple aspects as in conventional evaluation by observers. In addition, we also found that evaluation can be shown in a continuous scoring scope different from the five-point rating scale in conventional evaluation by observers. As only manual annotation is used in this study, it is believed that evaluation can be executed at an even lower cost by automated annotation in future studies.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Okada, S., et al.: Estimating communication skills using dialogue acts and nonverbal features in multiple discussion datasets. In: Proceedings of the 18th ACM International Conference on Multimodal Interaction, pp. 169–176. Association for Computing Machinery, New York (2016)
Avci, U., Aran, O.: Predicting the performance in decision-making tasks: from individual cues to group interaction. IEEE Trans. Multimed. 18, 643–658 (2016). https://doi.org/10.1109/TMM.2016.2521348
Murray, G., Oertel, C.: Predicting group performance in task-based interaction. In: Proceedings of the 20th ACM International Conference on Multimodal Interaction, pp. 14–20. Association for Computing Machinery, New York (2018)
Klaas, M.: Toward indicative discussion fora summarization. UBC CS TR-2005. 4 (2005)
Sharifi, B., Hutton, M.-A., Kalita, J.K.: Experiments in microblog summarization. In: 2010 IEEE Second International Conference on Social Computing, pp. 49–56. ieeexplore.ieee.org (2010)
Snow, E.L., Allen, L.K., Jacovina, M.E., Crossley, S.A., Perret, C.A., McNamara, D.S.: Keys to detecting writing flexibility over time: entropy and natural language processing. Learn. Anal. 2, 40–54 (2015). https://doi.org/10.18608/jla.2015.23.4
Ke, Z., Ng, V.: Automated essay scoring: a survey of the state of the art. In: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence Organization, California (2019)
Kunz, W., Rittel, H.W.J.: Issues as elements of information systems. Institute of Urban and Regional Development, University of California at Berkeley Working Paper 131, 14 (1970)
Conklin, J., Begeman, M.L.: GIBIS: a hypertext tool for exploratory policy discussion. ACM Trans. Inf. Syst. Secur. 6, 303–331 (1988). https://doi.org/10.1145/58566.59297
Burgess Yakemovic, K.C., Conklin, E.J.: Report on a development project use of an issue-based information system. In: Proceedings of the 1990 ACM Conference on Computer-Supported Cooperative Work, pp. 105–118. Association for Computing Machinery, New York (1990)
van Gelder, T.: Enhancing deliberation through computer supported argument visualization. In: Kirschner, P.A., Buckingham Shum, S.J., Carr, C.S. (eds.) Visualizing Argumentation: Software Tools for Collaborative and Educational Sense-Making, pp. 97–115. Springer, London, London (2003). https://doi.org/10.1007/978-1-4471-0037-9_5
Selvin, A., et al.: Compendium: making meetings into knowledge events (2001)
Barker, E., Gaizauskas, R.: Summarizing multi-party argumentative conversations in reader comment on news. In: Proceedings of the Third Workshop on Argument Mining (ArgMining2016), Berlin, Germany, pp. 12–20. Association for Computational Linguistics (2016)
Lawrence, J., Reed, C.: Argument mining: a survey. Comput. Linguist. Assoc. Comput. Linguist. 45, 765–818 (2020). https://doi.org/10.1162/coli_a_00364
Alsinet, T., Argelich, J., Béjar, R., Martínez, S.: Measuring user relevance in online debates through an argumentative model. Pattern Recogn. Lett. 133, 41–47 (2020). https://doi.org/10.1016/j.patrec.2020.02.008
Stab, C., Gurevych, I.: Annotating argument components and relations in persuasive essays. In: Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, Dublin, Ireland, pp. 1501–1510. Dublin City University and Association for Computational Linguistics (2014)
Stab, C., Gurevych, I.: Parsing argumentation structures in persuasive essays. Comput. Linguist. Assoc. Comput. Linguist. 43, 619–659 (2017). https://doi.org/10.1162/coli_a_00295
Sawyer, R.K.: Creative teaching: collaborative discussion as disciplined improvisation. Educ. Res. 33, 12–20 (2004). https://doi.org/10.3102/0013189X033002012
Alexandersson, J., et al.: Dialogue acts in VERBMOBIL-2 (1997)
Renals, S., Hain, T., Bourlard, H.: Recognition and understanding of meetings the AMI and AMIDA projects. In: 2007 IEEE Workshop on Automatic Speech Recognition & Understanding (ASRU), pp. 238–247 (2007)
Fang, A., Cao, J., Bunt, H., Liu, X.: The annotation of the Switchboard corpus with the new ISO standard for dialogue act analysis. In: Workshop on Interoperable Semantic Annotation, p. 13. sigsem.uvt.nl (2012)
Sinclair, J., Coulthard, M.: Towards an analysis of discourse. taylorfrancis.com (2013)
ISO 24617-2:2020. https://www.iso.org/standard/76443.html
Benne, K.D., Sheats, P.: Functional roles of group members. J. Soc. Issues 4, 41–49 (1948). https://doi.org/10.1111/j.1540-4560.1948.tb01783.x
Salazar, A.J.: An analysis of the development and evolution of roles in the small group. Small Group Res. 27, 475–503 (1996). https://doi.org/10.1177/1046496496274001
Pianesi, F., Zancanaro, M., Lepri, B., Cappelletti, A.: A multimodal annotated corpus of consensus decision making meetings. Lang. Resour. Eval. 41, 409–429 (2007). https://doi.org/10.1007/s10579-007-9060-6
Fujimoto, M.: Team roles and hierarchic system in group discussion. Group Decis. Negot. 25(3), 585–608 (2015). https://doi.org/10.1007/s10726-015-9453-7
Bartel, C.A., Saavedra, R.: The collective construction of work group moods. Adm. Sci. Q. 45, 197–231 (2000). https://doi.org/10.2307/2667070
Mihalcea, R., Tarau, P.: TextRank: bringing order into text. In: Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, Barcelona, Spain, pp. 404–411. Association for Computational Linguistics (2004)
Murakami, A., Nasukawai, T., Nakagawa, H.: Extracting useful remarks in online discussions. In: Proceedings of the Fourteenth Annual Meeting of the Association for Natural Language Processing, pp. 352–355 (2008). (in Japanese)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Masuda, T., Hirose, H., Tsuchiya, T. (2023). Structuring of Discourse and Annotation Method for Contribution Assessment in Collaborative Discussions. In: Dang, T.K., Küng, J., Chung, T.M. (eds) Future Data and Security Engineering. Big Data, Security and Privacy, Smart City and Industry 4.0 Applications. FDSE 2023. Communications in Computer and Information Science, vol 1925. Springer, Singapore. https://doi.org/10.1007/978-981-99-8296-7_6
Download citation
DOI: https://doi.org/10.1007/978-981-99-8296-7_6
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8295-0
Online ISBN: 978-981-99-8296-7
eBook Packages: Computer ScienceComputer Science (R0)