Back
Announcing RoastMyPost: LLMs Eval Blog Posts and More
webAuthor
Credibility Rating
3/5
Good(3)Good quality. Reputable source with community review or editorial standards, but less rigorous than peer-reviewed venues.
Rating inherited from publication venue: EA Forum
An EA Forum announcement for a practical LLM-based writing evaluation tool; tangentially relevant to AI safety through its demonstration of LLMs-as-evaluators, a technique also used in alignment research for scalable oversight and automated feedback.
Metadata
Importance: 28/100tool
Summary
RoastMyPost is a tool that uses large language models to evaluate and critique blog posts, particularly those submitted to forums like the EA Forum. It provides automated feedback on writing quality, argumentation, and clarity to help authors improve their posts before or after publication.
Key Points
- •Introduces RoastMyPost, an LLM-powered tool designed to evaluate and provide critical feedback on blog posts and forum submissions.
- •Leverages AI to assess writing quality, logical consistency, and argumentative strength in a 'roast' style critique format.
- •Targets the EA Forum community as a primary use case, helping authors refine ideas before wider dissemination.
- •Demonstrates a practical application of LLMs as evaluators of written content rather than just generators.
- •Raises questions about the reliability and calibration of AI-generated critique for nuanced intellectual writing.
Cached Content Preview
HTTP 200Fetched Apr 10, 202615 KB
# Announcing RoastMyPost: LLMs Eval Blog Posts and More
By Ozzie Gooen
Published: 2025-12-17
Today we're releasing [RoastMyPost](https://www.roastmypost.org/), a new experimental application for blog post evaluation using LLMs. [Try it Here](https://www.roastmypost.org/).
### TLDR
* [RoastMyPost](https://roastmypost.org) is a new [QURI](https://quantifieduncertainty.org/) application that uses LLMs and code to **evaluate blog posts and research documents.**
* It uses a variety of ***LLM evaluators***. Most are narrow checks: Fact Check, Spell Check, Fallacy Check, Math Check, Link Check, Forecast Check, and others.
* Optimized for **EA & Rationalist content** with direct import from EA Forum and LessWrong URLs. Other links use standard web fetching.
* Works best for **200 - ~10,000 word documents with factual assertions and simple formatting**. It can also do basic reviewing of [Squiggle models](https://www.roastmypost.org/docs/jqjlI1_CntHsTppG/reader). Longer documents and documents in LaTeX will experience slowdowns and errors.
* [**Open source**](https://github.com/quantified-uncertainty/roast-my-post), **free** for reasonable use\[1\]. Public examples are [here](https://www.roastmypost.org/docs).
* Experimentation encouraged! We're all figuring out how to best use these tools.
* Overall, we're most interested in using RoastMyPost as an experiment for potential LLM document workflows. The tech is early now, but it's at a good point for experimentation.
A representative illustration
How It Works
------------
1. Import a document. Submit markdown text or provide the URL of a publicly accessible post.
2. Select evaluators to run. A few are system-recommended. Others are custom evaluators submitted by users. Quality varies, so use with appropriate skepticism.
3. Wait 1-5 minutes for processing. (potentially more if the site is busy)
4. Review the results.
5. Add or re-run evaluations as needed.
Screenshots
-----------
**Reader Page**
The reader page is the [main article view](https://www.roastmypost.org/docs/sQcHHOZnVkdcz4xF2v10j/reader?evals=system-fact-checker%2Csystem-forecast-checker%2Csystem-link-verifier%2Csystem-math-checker%2Csystem-spelling-grammar%2Csystem-fallacy-check). You can toggle different evaluators, each has a different set of inline comments.
**Editor Page**
Add/remove/rerun evaluations and make other edits.
**Posts Page**
**Current AI Agents / Workflows**
---------------------------------
<table><tbody><tr><td style="border-color:#dddddd;padding
... (truncated, 15 KB total)Resource ID:
a49c607747375f68 | Stable ID: sid_6qlp3GbUvW