<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="dai-etal-2024-critical">
    <titleInfo>
        <title>A Critical Look at Meta-evaluating Summarisation Evaluation Metrics</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Xiang</namePart>
        <namePart type="family">Dai</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sarvnaz</namePart>
        <namePart type="family">Karimi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Biaoyan</namePart>
        <namePart type="family">Fang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2024</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yaser</namePart>
            <namePart type="family">Al-Onaizan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mohit</namePart>
            <namePart type="family">Bansal</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yun-Nung</namePart>
            <namePart type="family">Chen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Miami, Florida, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Effective summarisation evaluation metrics enable researchers and practitioners to compare different summarisation systems efficiently. Estimating the effectiveness of an automatic evaluation metric, termed meta-evaluation, is a critically important research question. In this position paper, we review recent meta-evaluation practices for summarisation evaluation metrics and find that (1) evaluation metrics are primarily meta-evaluated on datasets consisting of examples from news summarisation datasets, and (2) there has been a noticeable shift in research focus towards evaluating the faithfulness of generated summaries. We argue that the time is ripe to build more diverse benchmarks that enable the development of more robust evaluation metrics and analyze the generalization ability of existing evaluation metrics. In addition, we call for research focusing on user-centric quality dimensions that consider the generated summary‘s communicative goal and the role of summarisation in the workflow.</abstract>
    <identifier type="citekey">dai-etal-2024-critical</identifier>
    <identifier type="doi">10.18653/v1/2024.findings-emnlp.869</identifier>
    <location>
        <url>https://aclanthology.org/2024.findings-emnlp.869/</url>
    </location>
    <part>
        <date>2024-11</date>
        <extent unit="page">
            <start>14795</start>
            <end>14808</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T A Critical Look at Meta-evaluating Summarisation Evaluation Metrics
%A Dai, Xiang
%A Karimi, Sarvnaz
%A Fang, Biaoyan
%Y Al-Onaizan, Yaser
%Y Bansal, Mohit
%Y Chen, Yun-Nung
%S Findings of the Association for Computational Linguistics: EMNLP 2024
%D 2024
%8 November
%I Association for Computational Linguistics
%C Miami, Florida, USA
%F dai-etal-2024-critical
%X Effective summarisation evaluation metrics enable researchers and practitioners to compare different summarisation systems efficiently. Estimating the effectiveness of an automatic evaluation metric, termed meta-evaluation, is a critically important research question. In this position paper, we review recent meta-evaluation practices for summarisation evaluation metrics and find that (1) evaluation metrics are primarily meta-evaluated on datasets consisting of examples from news summarisation datasets, and (2) there has been a noticeable shift in research focus towards evaluating the faithfulness of generated summaries. We argue that the time is ripe to build more diverse benchmarks that enable the development of more robust evaluation metrics and analyze the generalization ability of existing evaluation metrics. In addition, we call for research focusing on user-centric quality dimensions that consider the generated summary‘s communicative goal and the role of summarisation in the workflow.
%R 10.18653/v1/2024.findings-emnlp.869
%U https://aclanthology.org/2024.findings-emnlp.869/
%U https://doi.org/10.18653/v1/2024.findings-emnlp.869
%P 14795-14808

Markdown (Informal)

[A Critical Look at Meta-evaluating Summarisation Evaluation Metrics](https://aclanthology.org/2024.findings-emnlp.869/) (Dai et al., Findings 2024)

• A Critical Look at Meta-evaluating Summarisation Evaluation Metrics (Dai et al., Findings 2024)

ACL

• Xiang Dai, Sarvnaz Karimi, and Biaoyan Fang. 2024. A Critical Look at Meta-evaluating Summarisation Evaluation Metrics. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 14795–14808, Miami, Florida, USA. Association for Computational Linguistics.