Abstract
The need for an evaluation method of spoken dialogue systems as a whole is more critical today than ever before. However, previous evaluation methods are no longer adequate for evaluating interactive dialogue systems. We have designed a new evaluation method that is system-to-system automatic dialogue with linguistic noise. By linguistic noise we simulate speech recognition errors in Spoken Dialogue Systems. Therefore, robustness of language understanding and of dialogue management can be evaluated. We have implemented an evaluation environment for automatic dialogue. We examined the validity of this method for automatic dialogue under different error rates and different dialogue strategies.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Araki, M., Kawahara, T. and Doshita, S.: A keyword-driven parser for spontaneous speech understanding. In Proc. Int'l Sympo. on Spoken Dialogue (1993) 113–116
Araki, M. and Doshita, S.: Cooperative spoken dialogue model using Bayesian network and event hierarchy. Trans. of IEICE, E78-d(6) (1995) 629–635
Carletta, J. C.: Risk-taking and Recovery in Task-Oriented Dialogue. PhD thesis, University of Edinburgh, (1992)
Grosz, B. J. and Sidner, C. L.: Plans for discourse. In Cohen, P. R., Morgan, J. and Pollack, M. E. editors, Intentions in Communication. The MIT Press, (1990) 417–444
Hashida, K. et al.: DiaLeague. In Proc. of the first annual meeting of the association for natural language processing (in Japanese) (1995) 309–312
Hirshman L.: Human language evaluation. In Proc. of ARPA Human Language Technology Workshop (1994) 99–101
Kautz, H. A.: A circumscriptive theory of plan recognition. In Cohen, P. R., Morgan, J. and Pollack, M. E. editors, Intentions in Communication. The MIT Press, (1990) 105–133
Moore, R. C.: Semantic evaluation for spoken-language systems. In Proc. of ARPA Human Language Technology Workshop (1994) 126–131
Pollack, M. E.: Plans as complex mental attitudes. In P. R. Cohen, J. Morgan, and M. E. Pollack, editors, Intentions in Communication. The MIT Press, (1990) 77–103
Vilain, M.: Getting serious about parsing plans: a grammatical analysis of plan recognition. In Proc. of AAAI (1990) 190–197
Walker, M. A.: Discourse and deliberation: Testing a collaborative strategy. In Proc. of COLING94 (1994) 1205–1211
Walker, M. A.: Experimentally evaluating communicative strategies: The effect of the task. In Proc. of AAAI94 (1994) 86–93
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 1997 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Araki, M., Doshita, S. (1997). Automatic evaluation environment for Spoken Dialogue Systems. In: Maier, E., Mast, M., LuperFoy, S. (eds) Dialogue Processing in Spoken Language Systems. DPSLS 1996. Lecture Notes in Computer Science, vol 1236. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-63175-5_46
Download citation
DOI: https://doi.org/10.1007/3-540-63175-5_46
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-63175-0
Online ISBN: 978-3-540-69206-5
eBook Packages: Springer Book Archive