When AI Models Go Rogue: Building AI You Can Trust, with Zico Kolter of Gray Swan

When AI Models Go Rogue: Building AI You Can Trust, with Zico Kolter of Gray Swan

Oct 9 · 25 min

People

  • Vivek Ramaswami Host
  • Zico Kolter Guest

Watch on

Suggest an edit

In this fireside chat, Gray Swan co-founder and chief scientist Zico Kolter, who also heads CMU’s machine learning department and chairs the safety and security committee on OpenAI’s board, explains why AI agents create a security problem that traditional cybersecurity cannot fully solve. He covers prompt injection, adversarial robustness, what “pacing the frontier” means, and why a real-time protection agent may be needed when patching is too slow. He closes with practical steps founders can take today, starting with red teaming the AI component of their own products. 00:00 - Why AI Agents Are Breaking Out Right Now 01:40 - Today's AI Models Are the Worst They Will Ever Be and What That Means for Security 03:43 - How Credible Is P Doom? Why Sci-Fi AI Risks Become Tomorrow's Real Concerns 05:46 - What "Pacing the Frontier" Actually Means for AI Labs 08:36 - What AI Alignment Really Means: Two Classes of AI Safety 10:08 - When AI Agents Overstep: Helpful Generalization vs. Going Rogue 12:44 - Prompt Injection Is a New Class of AI Exploit That Cybersecurity Alone Cannot Fix 15:29 - Founders: Red Team Your AI Systems, Not Just You Software 17:25 - Does Making AI Smarter Make It Safer? Capabilities vs. Robustness 20:15 - What Is AI Robustness? Performance Against an Adversary Explained 21:14 - Why Patching Fails Against AI Attackers and the Case for Real-Time AI Agent Watchdogs 23:26 - Zico Kolter's 12-Month Prediction: A Hardened AI and Cyber Ecosystem

Bay area Security How Credible Is P Worst They Will Ever Be OpenAI

Chapters

  1. 0:00 Why AI Agents Are Breaking Out Right Now
  2. 1:40 Today's AI Models Are the Worst They Will Ever Be and What That Means for Security
  3. 3:43 How Credible Is P Doom? Why Sci-Fi AI Risks Become Tomorrow's Real Concerns
  4. 5:46 What "Pacing the Frontier" Actually Means for AI Labs
  5. 8:36 What AI Alignment Really Means: Two Classes of AI Safety
  6. 10:08 When AI Agents Overstep: Helpful Generalization vs. Going Rogue
  7. 12:44 Prompt Injection Is a New Class of AI Exploit That Cybersecurity Alone Cannot Fix
  8. 15:29 Founders: Red Team Your AI Systems, Not Just You Software
  9. 17:25 Does Making AI Smarter Make It Safer? Capabilities vs. Robustness
  10. 20:15 What Is AI Robustness? Performance Against an Adversary Explained
  11. 21:14 Why Patching Fails Against AI Attackers and the Case for Real-Time AI Agent Watchdogs
  12. 23:26 Zico Kolter's 12-Month Prediction: A Hardened AI and Cyber Ecosystem

Related coverage

Conversation

No comments yet.