HalluLens, ACL 2025 (https://aclanthology.org/2025.acl-long.1176/)

HalluLens: LLM Hallucination Benchmark - ACL Anthology H allu L ens: LLM Hallucination Benchmark 

 Yejin Bang ,
 Ziwei Ji ,
 Alan Schelten ,
 Anthony Hartshorn ,
 Tara Fowler ,
 Cheng Zhang ,
 Nicola Cancedda ,
 Pascale Fung 

 Correct Metadata for 

 Use this form to create a GitHub issue with structured data describing the correction. You will need a GitHub account.
Once you create that issue, the correction will be reviewed by a staff member. ⚠️ Mobile Users: Submitting this form to create a new issue will only work with github.com, not the GitHub Mobile app. Important : The Anthology treat PDFs as authoritative. Please use this form only to correct data
that is out of line with the PDF. See our corrections
guidelines if you need to change the PDF. Title 
 Adjust the title. Retain tags such as
<fixed-case>. 
 Authors 
 Adjust author names and order to match the
PDF. Add Author Abstract 
 Correct abstract if needed. Retain XML formatting tags such as <tex-math>. You may use <b>...</b> for bold , <i>...</i> for italic , and <url>...</url> for URLs. 
 Verification against PDF 
 Ensure that the new title/authors match the snapshot below. (If there
is no snapshot or it is too small, consult the PDF .) Authors concatenated from the text boxes above: 
 ALL author names match the snapshot above—including
middle initials, hyphens, and accents. Create GitHub issue for staff review Abstract

 Large language models (LLMs) often generate responses that deviate from user input or training data, a phenomenon known as “hallucination.” These hallucinations undermine user trust and hinder the adoption of generative AI systems. Addressing hallucinations is important for the advancement of LLMs. This paper introduces a comprehensive hallucination benchmark HalluLens, incorporating both extrinsic and intrinsic evaluation tasks, built upon a clear taxonomy of hallucination. A major challenge in benchmarking hallucinations is the lack of a unified framework due to inconsistent definitions and categorizations. We disentangle LLM hallucination from “factuality” and propose a taxonomy distinguishing extrinsic and intrinsic hallucinations to promote consistency and facilitate research. We emphasize extrinsic hallucinations – where generated content deviates from training data – as they become increasingly relevant with LLM advancements. However, no benchmark is solely dedicated to extrinsic hallucinations. To address this gap, HalluLens introduces three new extrinsic tasks with dynamic test set generation to mitigate data leakage and ensure robustness. We release codebase for extrinsic hallucination benchmark. Anthology ID: 2025.acl-long.1176 Volume: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) Month: July Year: 2025 Address: Vienna, Austria Editors: Wanxiang Che ,
 Joyce Nabende ,
 Ekaterina Shutova ,
 Mohammad Taher Pilehvar Venue: ACL SIG: Publisher: Association for Computational Linguistics Note: Pa

... (truncated, 14 KB total)