Deepfake-Eval-2024 benchmark
paperAuthors
Credibility Rating
Good quality. Reputable source with community review or editorial standards, but less rigorous than peer-reviewed venues.
Rating inherited from publication venue: arXiv
Data Status
Abstract
In the age of increasingly realistic generative AI, robust deepfake detection is essential for mitigating fraud and disinformation. While many deepfake detectors report high accuracy on academic datasets, we show that these academic benchmarks are out of date and not representative of real-world deepfakes. We introduce Deepfake-Eval-2024, a new deepfake detection benchmark consisting of in-the-wild deepfakes collected from social media and deepfake detection platform users in 2024. Deepfake-Eval-2024 consists of 45 hours of videos, 56.5 hours of audio, and 1,975 images, encompassing the latest manipulation technologies. The benchmark contains diverse media content from 88 different websites in 52 different languages. We find that the performance of open-source state-of-the-art deepfake detection models drops precipitously when evaluated on Deepfake-Eval-2024, with AUC decreasing by 50% for video, 48% for audio, and 45% for image models compared to previous benchmarks. We also evaluate commercial deepfake detection models and models finetuned on Deepfake-Eval-2024, and find that they have superior performance to off-the-shelf open-source models, but do not yet reach the accuracy of deepfake forensic analysts. The dataset is available at https://github.com/nuriachandra/Deepfake-Eval-2024.
Cited by 2 pages
| Page | Type | Quality |
|---|---|---|
| Epistemic Collapse | Risk | 49.0 |
| AI-Powered Fraud | Risk | 69.0 |
Cached Content Preview
HTML conversions [sometimes display errors](https://info.dev.arxiv.org/about/accessibility_html_error_messages.html) due to content that did not convert correctly from the source. This paper uses the following packages that are not yet supported by the HTML conversion tool. Feedback on these issues are not necessary; they are known and are being worked on.
- failed: changepage
Authors: achieve the best HTML results from your LaTeX submissions by following these [best practices](https://info.arxiv.org/help/submit_latex_best_practices.html).
[License: CC BY-SA 4.0](https://info.arxiv.org/help/license/index.html#licenses-available)
arXiv:2503.02857v2 \[cs.CV\] 05 Mar 2025
# Deepfake-Eval-2024: A Multi-Modal In-the-Wild Benchmark of Deepfakes Circulated in 2024
Report issue for preceding element
Nuria Alina Chandra
TrueMedia.org
Ryan Murtfeldt
TrueMedia.org
University of Washington, Seattle
Lin Qiu
TrueMedia.org
University of Washington, Seattle
Arnab Karmakar
TrueMedia.org
University of Washington, Seattle
Hannah Lee
TrueMedia.org
Emmanuel Tanumihardja
TrueMedia.org
University of Washington, Seattle
Kevin Farhat
TrueMedia.org
University of Washington, Seattle
Ben Caffee
TrueMedia.org
University of Washington, Seattle
Sejin Paik
TrueMedia.org
Georgetown University, Washington D.C.
Changyeon Lee
Miraflow AI
Yonsei University, Seoul
Jongwook Choi
TrueMedia.org
Chung-Ang University, Seoul
Aerin Kim
TrueMedia.org
Miraflow AI
Oren Etzioni
TrueMedia.org
University of Washington, Seattle
Report issue for preceding element
###### Abstract
Report issue for preceding element
In the age of increasingly realistic generative AI, robust deepfake detection is essential for mitigating fraud and disinformation. While many deepfake detectors report high accuracy on academic datasets, we show that these academic benchmarks are out of date and not representative of real-world deepfakes. We introduce Deepfake-Eval-2024, a new deepfake detection benchmark consisting of in-the-wild deepfakes collected from social media and deepfake detection platform users in 2024. Deepfake-Eval-2024 consists of 45 hours of videos, 56.5 hours of audio, and 1,975 images, encompassing the latest manipulation technologies. The benchmark contains diverse media content from 88 different websites in 52 different languages. We find that the performance of open-source state-of-the-art deepfake detection models drops precipitously when evaluated on Deepfake-Eval-2024, with AUC decreasing by 50% for video, 48% for audio, and 45% for image models compared to previous benchmarks. We also evaluate commercial deepfake detection models and models finetuned on Deepfake-Eval-2024, and find that they have superior performance to off-the-shelf open-source models, but do not yet reach the accuracy of deepfake forensic analysts. The dataset is available at https://github.com/nuriachandra/Deepfake-Eval-2024.
Report issue for preceding element
f39c2cc4c0f303cc | Stable ID: YTNmMjNjY2