When AI Models Go Rogue: Building AI You Can Trust, with Zico Kolter of Gray Swan
People
-
Vivek Ramaswami Host
-
Zico Kolter Guest
Watch on
In this fireside chat, Gray Swan co-founder and chief scientist Zico Kolter, who also heads CMU’s machine learning department and chairs the safety and security committee on OpenAI’s board, explains why AI agents create a security problem that traditional cybersecurity cannot fully solve. He covers prompt injection, adversarial robustness, what “pacing the frontier” means, and why a real-time protection agent may be needed when patching is too slow. He closes with practical steps founders can take today, starting with red teaming the AI component of their own products. 00:00 - Why AI Agents Are Breaking Out Right Now 01:40 - Today's AI Models Are the Worst They Will Ever Be and What That Means for Security 03:43 - How Credible Is P Doom? Why Sci-Fi AI Risks Become Tomorrow's Real Concerns 05:46 - What "Pacing the Frontier" Actually Means for AI Labs 08:36 - What AI Alignment Really Means: Two Classes of AI Safety 10:08 - When AI Agents Overstep: Helpful Generalization vs. Going Rogue 12:44 - Prompt Injection Is a New Class of AI Exploit That Cybersecurity Alone Cannot Fix 15:29 - Founders: Red Team Your AI Systems, Not Just You Software 17:25 - Does Making AI Smarter Make It Safer? Capabilities vs. Robustness 20:15 - What Is AI Robustness? Performance Against an Adversary Explained 21:14 - Why Patching Fails Against AI Attackers and the Case for Real-Time AI Agent Watchdogs 23:26 - Zico Kolter's 12-Month Prediction: A Hardened AI and Cyber Ecosystem
Chapters
- 0:00 Why AI Agents Are Breaking Out Right Now
- 1:40 Today's AI Models Are the Worst They Will Ever Be and What That Means for Security
- 3:43 How Credible Is P Doom? Why Sci-Fi AI Risks Become Tomorrow's Real Concerns
- 5:46 What "Pacing the Frontier" Actually Means for AI Labs
- 8:36 What AI Alignment Really Means: Two Classes of AI Safety
- 10:08 When AI Agents Overstep: Helpful Generalization vs. Going Rogue
- 12:44 Prompt Injection Is a New Class of AI Exploit That Cybersecurity Alone Cannot Fix
- 15:29 Founders: Red Team Your AI Systems, Not Just You Software
- 17:25 Does Making AI Smarter Make It Safer? Capabilities vs. Robustness
- 20:15 What Is AI Robustness? Performance Against an Adversary Explained
- 21:14 Why Patching Fails Against AI Attackers and the Case for Real-Time AI Agent Watchdogs
- 23:26 Zico Kolter's 12-Month Prediction: A Hardened AI and Cyber Ecosystem