Critical Insights
A Critical Insight is a piece of knowledge that is surprising, important, and compact. This framework helps identify and prioritize the highest-value contributions to understanding—the kind of insights that genuinely shift how informed people think about a problem.
The Three Criteria
Section titled “The Three Criteria”| Criterion | Definition | Test |
|---|---|---|
| Surprising | Updates beliefs of informed people | Would an AI safety researcher’s credences shift after learning this? |
| Important | Affects high-stakes decisions | Does this change what funders/researchers should prioritize? |
| Compact | Claim + key evidence fits briefly | Can you convey the core insight and its justification in a few sentences? |
Surprising
Section titled “Surprising”An insight is surprising if learning it would cause an informed person to update their beliefs. This isn’t about being novel to complete beginners—it’s about genuinely new information or synthesis that even domain experts haven’t fully internalized.
Examples:
- “AI labs spend more on lobbying than safety research” (if true and documented)
- “The top 3 interpretability researchers all left academia for industry in 2024”
- A rigorous estimate that contradicts conventional wisdom
Non-examples:
- “AI capabilities are advancing quickly” (widely known)
- “Alignment is hard” (already consensus among target audience)
Important
Section titled “Important”An insight is important if it affects decisions with significant consequences. In the AI safety context, this means influencing:
- Where funding should go
- What research directions to prioritize
- Which policy interventions to pursue
- How to assess different risk scenarios
Importance can be estimated roughly by asking: “If key decision-makers believed this, would their actions change?”
Compact
Section titled “Compact”The compactness criterion is not about brevity for its own sake—it’s about transmissibility and verifiability. An insight that requires reading thousands of pages to appreciate isn’t actionable for most decision-makers.
The claim and its main evidence should fit in a short paragraph. This doesn’t mean the full analysis must be brief—supporting details, methodology, and caveats can be extensive—but the core contribution should be extractable.
Theoretical Foundations
Section titled “Theoretical Foundations”The Critical Insight framework draws on concepts from multiple disciplines.
Information Theory: Surprisal
Section titled “Information Theory: Surprisal”Claude Shannon’s information theory formalized the intuition that information content is inversely related to probability. The “surprisal” of an event is measured as:
Information (bits) = -log₂(probability)A fair coin landing heads carries 1 bit of information. A biased coin (90% heads) landing heads carries only 0.15 bits—because it’s less surprising.
Application to insights: A claim that most informed people already believe (high prior probability) carries little information. The goal is to find claims that are both true and would update priors significantly.
Machine Learning: Information Gain
Section titled “Machine Learning: Information Gain”In decision tree algorithms, information gain measures how much a feature reduces uncertainty about a target variable. Features are selected based on their ability to split the data in informative ways.
Application to insights: The most valuable research questions are those whose answers would most reduce uncertainty about important decisions—analogous to selecting the highest-information-gain feature.
Statistics: Parsimony and Model Selection
Section titled “Statistics: Parsimony and Model Selection”The Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC) formalize Occam’s razor: models should balance fit against complexity. Both penalize additional parameters.
“All things equal, we should prefer the simpler model over any more complex alternative. This is known as the ‘principle of parsimony.’”
Application to insights: Compact insights are preferable because they’re easier to verify, communicate, and act upon. A theory that explains the same phenomena with fewer moving parts is more robust.
Decision Theory: Value of Information
Section titled “Decision Theory: Value of Information”Value of Information (VOI) analysis provides a formal framework for research prioritization:
“VOI quantifies the expected benefit from acquiring additional evidence to inform a decision. The value of a study is the extent to which it reduces uncertainty, thus potentially reducing errors in decision-making.”
Key VOI concepts:
- Expected Value of Perfect Information (EVPI): The maximum you should pay to eliminate all uncertainty
- Expected Value of Partial Perfect Information (EVPPI): Value of resolving uncertainty on specific parameters
- Expected Value of Sample Information (EVSI): Value of a specific study with realistic sample sizes
Application to insights: The “importance” criterion corresponds to high VOI—insights that would significantly change optimal decisions if known.
Research Debt and Distillation
Section titled “Research Debt and Distillation”Distill.pub’s concept of Research Debt identifies a systematic underinvestment in explanation:
“Research debt is the accumulation of missing interpretive labor. The maintainable size of a field is controlled by how members trade off energy between communicating and understanding.”
The economics favor explanation: in one-to-many communication, the cost of explaining is O(1) while the cost of understanding across N people is O(N). Yet incentives typically reward producing new results over clarifying existing ones.
Application to insights: Compact insights reduce research debt. They make knowledge more transmissible, expanding the “carrying capacity” of a field.
Practical Example: Fermi Model Competition
Section titled “Practical Example: Fermi Model Competition”The Fermi Model Competition on LessWrong illustrates similar principles. Entries are judged on:
| Criterion | Weight | Description |
|---|---|---|
| Surprise | 40% | How much does learning the answer update your views? |
| Topic Relevance | 20% | Is the question decision-relevant? |
| Robustness | 20% | How confident are we in the result? |
| Model Quality | 20% | Is the analysis well-constructed? |
Note that surprise carries the highest weight—twice that of any other criterion. This reflects the insight that novelty (in the information-theoretic sense) is the scarcest ingredient.
Application to LongtermWiki
Section titled “Application to LongtermWiki”LongtermWiki content can be evaluated against the Critical Insight framework:
| Content Type | How to Apply Framework |
|---|---|
| Knowledge Base pages | Does this synthesize information in ways that would update an AI safety researcher? |
| Crux Maps | Are the identified cruxes genuinely decision-relevant, or just interesting? |
| Estimates | Would this quantification surprise informed readers? |
| Research Reports | Can the key findings be stated compactly? |
Prioritizing Content Creation
Section titled “Prioritizing Content Creation”When deciding what to work on:
- Surprising + Important + Compact → Highest priority
- Surprising + Important → Worth pursuing if compactness can be achieved
- Important but not surprising → Reference material (useful but not differentiated)
- Surprising but not important → Interesting trivia (low priority)
Testing for Critical Insights
Section titled “Testing for Critical Insights”Opinion Fuzzing can operationalize the “surprising” criterion:
- Present the insight to LLMs or informed readers
- Measure whether their stated credences shift
- Track which claims actually update beliefs vs. which just restate conventional wisdom
Related Concepts
Section titled “Related Concepts”| Concept | Relationship to Critical Insights |
|---|---|
| Cruxes | Cruxes are belief differences; critical insights resolve them |
| Value of Information | Formal measure of what “important” means |
| Bits of evidence | Information-theoretic measure of “surprising” |
| Minimum Description Length | Formal approach to “compact” |
| Research distillation | Process of making insights more compact |
Sources
Section titled “Sources”Information Theory
Section titled “Information Theory”- Information Theory - Wikipedia
- Information Content - Wikipedia
- Information is Surprise - Plus Magazine
Machine Learning
Section titled “Machine Learning”Model Selection
Section titled “Model Selection”Decision Theory
Section titled “Decision Theory”- Value of Information: A Tool to Improve Research Prioritization - PLOS Medicine
- Value of Information Analysis for Research Decisions - Value in Health