ArXiv preprint on Community Notes helpfulness rates and timing.
paperAuthors
Credibility Rating
Good quality. Reputable source with community review or editorial standards, but less rigorous than peer-reviewed venues.
Rating inherited from publication venue: arXiv
Relevant to AI safety discussions around scalable content moderation and human oversight mechanisms; Community Notes represents a real-world crowdsourced alignment/moderation system whose sustainability challenges inform debates about human-in-the-loop governance at scale.
Paper Details
Metadata
Abstract
Community Notes are emerging as an important option for content moderation. The Community Notes system pioneered by Twitter, now known as X, uses a bridging algorithm to identify user-generated context with upvotes across political divides, supposedly spinning consensual gold from partisan straw. It is important to understand the nature of the community behind Community Notes, especially as the feature has now been imitated by several billion-user platforms. We look for signs of stability and disruption in the X Community Notes community and interrogate the motivations other than partisan animus (Allen, Martel, and Rand 2022) which may be driving users to contribute. We conduct a novel analysis of the impact of having a note published, which requires being considered "helpful" by the bridging algorithm, utilising a regression discontinuity design. This allows stronger causal inference than conventional methods used with observational data. Our analysis shows the positive effect on future note authoring of having a note published. This highlights the risk of the current system, where the proportion of notes considered "helpful" (and therefore shown to users on X) is low, 10%, and declining. This analysis has implications for the future of Community Notes on X and the extension of this approach to other platforms.
Summary
This paper investigates the sustainability of X's Community Notes crowdsourced moderation system, finding that note publication significantly boosts future contributor activity but that only ~10% of notes are rated 'helpful'—a rate that is declining. The authors argue this low and falling helpfulness rate threatens contributor motivation and the long-term viability of bridging-based content moderation. The findings have implications for other platforms adopting similar systems.
Key Points
- •Using regression discontinuity design, the study shows that having a note published causally increases a contributor's future note-writing activity.
- •Only ~10% of submitted Community Notes are deemed 'helpful' and displayed to users, and this helpfulness rate is declining over time.
- •The low helpfulness rate risks demotivating contributors, threatening the system's long-term sustainability.
- •Community Notes uses a bridging algorithm requiring cross-partisan agreement, making helpfulness harder to achieve as the contributor base diversifies.
- •Findings are relevant to other major platforms (e.g., Meta) that are adopting or considering similar crowdsourced moderation mechanisms.
Cited by 2 pages
| Page | Type | Quality |
|---|---|---|
| X Community Notes | Project | 54.0 |
| X.com Platform Epistemics | Approach | 20.0 |
Cached Content Preview
Threats to the sustainability of Community Notes on X
Threats to the sustainability of Community Notes on X
Zahra Arjmandi-Lari 1 ,
Alexios Mantzarlis 2 ,
Tom Stafford 3
The authors assert a Creative Commons Attribution (CC BY 4.0) License for this preprint
Abstract
Community Notes are emerging as an important option for content moderation. The Community Notes system pioneered by Twitter, now known as X, uses a bridging algorithm to identify user-generated context with upvotes across political divides, supposedly spinning consensual gold from partisan straw. It is important to understand the nature of the community behind Community Notes, especially as the feature has now been imitated by several billion-user platforms. We look for signs of stability and disruption in the X Community Notes community and interrogate the motivations other than partisan animus (Allen, Martel, and Rand 2022 ) which may be driving users to contribute. We conduct a novel analysis of the impact of having a note published, which requires being considered “helpful” by the bridging algorithm, utilising a regression discontinuity design. This allows stronger causal inference than conventional methods used with observational data. Our analysis shows the positive effect on future note authoring of having a note published. This highlights the risk of the current system, where the proportion of notes considered “helpful” (and therefore shown to users on X) is low, 10%, and declining. This analysis has implications for the future of Community Notes on X and the extension of this approach to other platforms.
1 Introduction
Content moderation is an essential function of a digital platform, yet it is also a highly disputed one (Gillespie 2018 ) . Top-down decisions to remove or label (or not) a piece of content have led to advertiser boycotts, consumer complaints, and political pressure. While large online platforms have extensive policy guidelines describing violative behavior, the diversity of the speech they govern makes it nearly impossible to apply those policies completely consistently, and their scale means that even a small percentage of errors can affect thousands or millions of users.
Moderation decisions can be particularly challenging when it comes to content that is harder to define, such as misinformation. There is at least a baseline of consensus around what constitutes sexual or violent content; misinformation is by definition contextual and requires high-quality contradictory evidence to be available. Still, the public (Ejaz et al. 2024 ) and regulators (Husovec 2024 ) expect action from platforms in this realm, and unchecked online misinformation risks negatively affecting outcomes from health behaviors (Allen, Watts, and Rand 2024 ) to political polarization (Budak et al. 2024 ) .
This context informed the decision by X (then Twitter) to launch Birdwatch in January 2021 ( Twitter ; Wojcik et al. 2022
... (truncated, 27 KB total)c64c6bdd2df72736 | Stable ID: sid_gac35dZIyX