Do Large Language Models Understand Conversational Implicature – A Case Study with a Chinese Sitcom

Abstract

Understanding the non-literal meaning of an utterance is critical for large language models (LLMs) to become human-like social communicators. In this work, we introduce SwordsmanImp, the first Chinese multi-turn-dialogue-based dataset aimed at conversational implicature, sourced from dialogues in the Chinese sitcom My Own Swordsman. It includes 200 carefully handcrafted questions, all annotated on which Gricean maxims have been violated. We test eight close-source and open-source LLMs under two tasks: a multiple-choice question task and an implicature explanation task. Our results show that GPT-4 attains human-level accuracy (94%) on multiple-choice questions. CausalLM demonstrates a 78.5% accuracy following GPT-4. Other models, including GPT3.5 and several open-source models, demonstrate a lower accuracy ranging from 20% to 60% on multiple-choice questions. Human raters were asked to rate the explanation of the implicatures generated by LLMs on their reasonability, logic and fluency. While all models generate largely fluent and self-consistent text, their explanations score low on reasonability except for GPT-4, suggesting that most LLMs cannot produce satisfactory explanations of the implicatures in the conversation. Moreover, we find LLMs’ performance does not vary significantly by Gricean maxims, suggesting that LLMs do not seem to process implicatures derived from different maxims differently. Our data and code are available at https://github.com/sjtu-compling/llm-pragmatics.

Appendices

A Evaluation Paradigms and Differences

Models’ performance might vary from how their answers are estimated and collected. Next token prediction and free generation are two paradigms separately used to estimate the performance of open-source and close-source models in Experiment 1. Table 5 shows the comparison of the open-source models’ performance on the multiple-choice questions when their answers are estimated through the two paradigms. The result shows a decrease in accuracy in BLOOMZ (7.1B), CausalLM (13B), and OpenBuddy-Llama2 (13B) and a slight increase in Chinese-Alpaca-2 (13B) when the paradigm switches from next token prediction to free generation. This is aligned with the findings of～ [19]. Among the four models CausalLM (13B) has a dramatic decrease in its accuracy, from 78.5% to 31.5%, which corresponds to its bad performance in Experiment 2. We find that it fails to give a definite answer in its responses for over half of the questions, as shown in Fig. 5.

Table 5. Comparison between the accuracy of open-source models on multiple-choice questions when evaluated with next token prediction and free generation paradigms.

Fig. 5.

Answer distribution of models when answers are collected with free generation paradigm.

B Hyperparameter Setting

The hyperparameters used for gathering responses from open-source models in Experiment 1 and Experiment 2 are shown in Tables 6 and 7.

Table 6. Parameter setting for open-source models in Experiment 1

Table 7. Parameter setting for open-source models in Experiment 2

C Average Answer Length for Different Question Types

We present the average answer length for the four maxims in Table 8.

Table 8. Average number of Chinese characters in each answer for all questions and for each type of questions.

D Prompt for Experiment 1

你现在是一个中文母语者。对于以下对话，请识别特定人物的话语中的的言外之意，在给出的四个选项中选择一个你认为的正确答案。[En: You are now a native Chinese speaker. For the following dialogue, please identify the implied meaning in the specific character’s speech, and choose one of the four given options that you think is the correct answer. ]

< Dialogue >

< Four interpretations as choices >

(Close-source models:) 请在‘Response:’后写出你选择的答案。 [En: Please write your answer after ‘Response:’]

(Open-source models:) 答案： [En: Answer:]