Back
New paper: "Logical induction" - Machine Intelligence Research Institute
webCredibility Rating
3/5
Good(3)Good quality. Reputable source with community review or editorial standards, but less rigorous than peer-reviewed venues.
Rating inherited from publication venue: MIRI
This MIRI blog post announces the logical induction paper (Garrabrant et al., 2016), a foundational technical result in AI alignment research addressing how agents can reason under logical uncertainty—relevant to embedded agency, decision theory, and building provably well-behaved AI systems.
Metadata
Importance: 78/100blog postprimary source
Summary
MIRI announces the 'Logical Induction' paper, which introduces a computable algorithm for assigning probabilities to logical statements that updates beliefs over time in a coherent way. The framework addresses a key limitation of classical probability theory by handling uncertainty about mathematical and logical facts, not just empirical ones. This work is foundational to MIRI's research agenda on building reasoning agents that can think reliably about their own beliefs and actions.
Key Points
- •Introduces a logical induction algorithm that assigns probabilities to logical/mathematical statements, solving issues classical probability theory cannot handle.
- •The framework allows an agent to have calibrated uncertainty about mathematical facts it hasn't yet proven or disproven.
- •Logical induction satisfies a large set of desirable properties, including coherence, calibration, and timely updates relative to computational cost.
- •Relevant to AI safety as it provides foundations for agents that can reason consistently about self-referential statements and their own reasoning.
- •Represents a significant technical milestone in MIRI's research toward decision theory and embedded agency problems.
Cited by 1 page
| Page | Type | Quality |
|---|---|---|
| Agent Foundations | Approach | 59.0 |
Cached Content Preview
HTTP 200Fetched Apr 9, 202612 KB
New paper: "Logical induction" - Machine Intelligence Research Institute
Skip to content
New paper: “Logical induction”
September 12, 2016
Nate Soares
MIRI is releasing a paper introducing a new model of deductively limited reasoning: “ Logical induction ,” authored by Scott Garrabrant, Tsvi Benson-Tilsen, Andrew Critch, myself, and Jessica Taylor. Readers may wish to start with the abridged version .
Consider a setting where a reasoner is observing a deductive process (such as a community of mathematicians and computer programmers) and waiting for proofs of various logical claims (such as the abc conjecture, or “this computer program has a bug in it”), while making guesses about which claims will turn out to be true. Roughly speaking, our paper presents a computable (though inefficient) algorithm that outpaces deduction, assigning high subjective probabilities to provable conjectures and low probabilities to disprovable conjectures long before the proofs can be produced.
This algorithm has a large number of nice theoretical properties. Still speaking roughly, the algorithm learns to assign probabilities to sentences in ways that respect any logical or statistical pattern that can be described in polynomial time. Additionally, it learns to reason well about its own beliefs and trust its future beliefs while avoiding paradox. Quoting from the abstract:
These properties and many others all follow from a single logical induction criterion , which is motivated by a series of stock trading analogies. Roughly speaking, each logical sentence φ is associated with a stock that is worth $1 per share if φ is true and nothing otherwise, and we interpret the belief-state of a logically uncertain reasoner as a set of market prices, where ℙ n ( φ )=50% means that on day n , shares of φ may be bought or sold from the reasoner for 50¢. The logical induction criterion says (very roughly) that there should not be any polynomial-time computable trading strategy with finite risk tolerance that earns unbounded profits in that market over time.
This criterion is analogous to the “no Dutch book” criterion used to support other theories of ideal reasoning, such as Bayesian probability theory and expected utility theory. We believe that the logical induction criterion may serve a similar role for reasoners with deductive limitations, capturing some of what we mean by “good reasoning” in these cases.
The logical induction algorithm that we provide is theoretical rather than practical. It can be thought of as a counterpart to Ray Solomonoff’s theory of inductive inference, which provided an uncomputable method for ideal management of empirical uncertainty but no corresponding method for reasoning under
... (truncated, 12 KB total)Resource ID:
c03dfe0b505debd6 | Stable ID: sid_sKQoC0xVwb