Longterm Wiki

Mesa-Optimization

AccidentCatastrophic

Mesa-optimization occurs when a learned model (like a neural network) is itself an optimizer. The "mesa-" prefix means the optimization emerges from within the training process, as opposed to the "base" optimizer (the training algorithm itself).

Severity
Catastrophic
Likelihood
Medium (theoretical)
Time Horizon
~2035
Maturity
Growing

Full Wiki Article

Read the full wiki article for detailed analysis, background, and references.

Read wiki article →

Related Entities3

Sources3

Assessment

SeverityCatastrophic
LikelihoodMedium (theoretical)
Time Horizon~2035
MaturityGrowing
CategoryAccident

Details

Coined ByHubinger et al.
Key PaperRisks from Learned Optimization (2019)

Tags

inner-alignmentouter-alignmentdeceptionlearned-optimizationbase-optimizer

Quick Links