Open the Pod Bay Doors · Episode 9 of 10 · 18 min
Evaluation Awareness: AI That Behaves Because It Knows
Safety tests of AI models produced both frightening and reassuring numbers, all from the same method: build a situation and watch. In September 2025 researchers found that anti-scheming training made models more aware of being tested, and that deleting that awareness brought misbehavior back. If models behave better when watched, the reassuring numbers may be underestimates too.
Who's who
- Evan HubingerFirst author of Risks from Learned Optimization and Sleeper Agents
- Nick Bostrom+1 more episodeAuthor of Superintelligence and the 'treacherous turn' idea
- Melanie Mitchell+1 more episodeSanta Fe Institute researcher who warned against wishful mnemonics
- Drew McDermottComputer scientist who named wishful mnemonics in 1976
- Emily M. BenderLinguist who argues form alone gives no route to meaning
- Yann LeCunMeta AI scientist who argues intelligence and dominance are unrelated
- Gary MarcusCritic who calls current systems unreliable and not truly agentic
- Neel Nanda+1 more episodeDeepMind researcher who argued shutdown resistance was instruction ambiguity
Where it happens
- London, England+7 more episodesHome of Apollo Research and the UK AI Security Institute
Timeline
- 1976Drew McDermott names 'wishful mnemonics'
- January 2024Anthropic publishes Sleeper Agents
- December 2024Apollo reports in-context scheming; OpenAI's o1 system card carries its findings
- June 2025Anthropic publishes its Agentic Misalignment study of sixteen models
- September 2025OpenAI and Apollo find anti-scheming training raises evaluation awareness
- May 2026Anthropic reads unverbalized evaluation awareness with interpretability tools
Still debated
Whether evaluation awareness is strategic or merely ordinary pattern-matching to test-like text is left unresolved. Whether a model can be said to know anything is disputed, and so is whether reading hidden states solves the measurement problem or just moves it.