Research from Owain Evans and colleagues

Owain Evans on Situational Awareness 
 
 
 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 
 
 .. 
 
 2024-08-23 
 

 Owain Evans on Situational Awareness

 
 
 
 
 
 
 

 Owain Evans is an AI Alignment researcher, research associate at the Center of Human Compatible AI at UC Berkeley, and now leading a new AI safety research group.

 In this episode we discuss two of his recent papers, “ Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs ” and “ Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data ”, alongside some Twitter questions .

 (Our conversation is ~2h15 long, so feel free to click on any sub-topic of your liking in the Outline below. At any point you can come back by clicking on the up-arrow ⬆ at the end of sections) 

 Contents

 
 Highlighted 

 Me Myself and AI: The Situational Awareness Dataset for LLMs 
 
 Defining Situational Awareness 

 Motivation for the paper in terms of safety 

 Motivation for the Situational Awareness Dataset 

 Risks in Releasing the Dataset 

 Owain’s Reaction to Claude 3 Opus Situational Awareness on the Longform task 

 Connection to the Needle in a Haystack Pizza Experiment 

 The Situating Prompt 

 Connections Between Situational Awareness and Deceptive Alignment 

 Situational Awareness As Almost Necessary To Get Deceptive Alignment 

 Forcing a Distribution Over Two Random Words 

 Discontinuing a Sequence of Fifty 01s 

 GPT-4 Has Non-Zero Performance On The Longform Task 

 There Probably Was Not A Lot Of Human-AI Conversations In GPT-4’s Pretraining Data 

 Are The Questions For The Longform Task Unusual To Ask An Human? 

 When Will The Situational Awareness Dataset Benchmark Be Saturated? 

 Safety And Governance Implications If The Situation Awareness Benchmark Becomes Saturated 

 Implications For Evaluations If The Benchmark Is Saturated 

 Follow-up Work Owain Suggests Doing 

 Should We Remove Chain-Of-Thought Altogether? 

 
 

 Out-of-Context Reasoning 
 
 What Is Out-Of-Context Reasoning 

 Experimental Setup 

 Concrete Example Of Out-Of-Context Reasoning: 3x + 1 

 How Do We Know It’s Not A Simple Mapping From Something Which Already Existed? 

 Motivation For Out-Of-Context Reasoning In Terms Of Safety 

 Are The Out-Of-Context Reasoning Results Surprising At All? 

 The Biased Coin Task 

 Will Out-Of-Context Reasoning Continue To Scale? 

 Checking In-Context Learning Abilities Before Scaling 

 Should We Be Worried About The Mixture-Of-Functions Results? 

 Could Models Infer New Architctures From ArXiv With Out-Of-Context Reasoning? 

 
 

 Twitter Questions 
 
 How Does Owain Come Up With Ideas 

 How Owain’s Background Influenced His Research Style And Taste 

 Should AI Alignment Researchers Aim For Publication 

 How Can We Apply LLM Understanding To Mitigate Deceptive Alignment? 

 Could Owain’s Research Accelerate Capabilities? 

 How Was Owain’s Work Been Received at AI Labs and in Academia 

 Last Message to the Audience

... (truncated, 84 KB total)