MAFALDA: A Benchmark and Comprehensive Study of Fallacy Detection and Classification - Equipe Data, Intelligence and Graphs
Communication Dans Un Congrès Année : 2024

MAFALDA: A Benchmark and Comprehensive Study of Fallacy Detection and Classification

Résumé

We introduce MAFALDA, a benchmark for fallacy classification that merges and unites previous fallacy datasets. It comes with a taxonomy that aligns, refines, and unifies existing classifications of fallacies. We further provide a manual annotation of a part of the dataset together with manual explanations for each annotation. We propose a new annotation scheme tailored for subjective NLP tasks, and a new evaluation method designed to handle subjectivity. We then evaluate several language models under a zero-shot learning setting and human performances on MAFALDA to assess their capability to detect and classify fallacies.
Fichier principal
Vignette du fichier
2024.naacl-long.270.pdf (1.03 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04631163 , version 1 (01-07-2024)

Licence

Identifiants

  • HAL Id : hal-04631163 , version 1

Citer

Chadi Helwe, Tom Calamai, Pierre-Henri Paris, Chloé Clavel, Fabian M Suchanek. MAFALDA: A Benchmark and Comprehensive Study of Fallacy Detection and Classification. NAACL 2024 - North American Chapter of the Association for Computational Linguistics, Jun 2024, Mexico City, Mexico. ⟨hal-04631163⟩
311 Consultations
50 Téléchargements

Partager

More