The Factuality of Large Language Models in the Legal Domain - Département Informatique et Réseaux
Conference Papers Year : 2024

The Factuality of Large Language Models in the Legal Domain

Abstract

This paper investigates the factuality of large language models (LLMs) as knowledge bases in the legal domain, in a realistic usage scenario: we allow for acceptable variations in the answer, and let the model abstain from answering when uncertain. First, we design a dataset of diverse factual questions about case law and legislation. We then use the dataset to evaluate several LLMs under different evaluation methods, including exact, alias, and fuzzy matching. Our results show that the performance improves significantly under the alias and fuzzy matching methods. Further, we explore the impact of abstaining and in-context examples, finding that both strategies enhance precision. Finally, we demonstrate that additional pre-training on legal documents, as seen with SaulLM, further improves factual precision from 63% to 81%.
Fichier principal
Vignette du fichier
factuality_legal_LLM.pdf (185.43 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-04701151 , version 1 (18-09-2024)

Licence

Identifiers

  • HAL Id : hal-04701151 , version 1

Cite

Rajaa El Hamdani, Thomas Bonald, Fragkiskos D. Malliaros, Nils Holzenberger, Fabian M. Suchanek. The Factuality of Large Language Models in the Legal Domain. CIKM 2024 - 33rd ACM International Conference on Information and Knowledge Management, Oct 2024, Boise Idaho, United States. ⟨hal-04701151⟩
130 View
20 Download

Share

More