Why GPUs Aren't the Answer for Every AI Workload | Sid Sheth Founder and CEO of d-Matrix

Why GPUs Aren't the Answer for Every AI Workload | Sid Sheth Founder and CEO of d-Matrix

Nov 2 · 48 min

People

  • Nataraj Host
  • Sid Sheth Guest

Listen on

Suggest an edit

Sid Sheth, co-founder and CEO of D-Matrix, joins Nataraj on Startup Project to talk about the future of AI inference, why GPUs may not be the answer for every workload, and what it takes to build a semiconductor startup.Key takeaways Training and inference are like education and a career. Sid's view is that inference is the much larger opportunity. D-Matrix focuses on ultra-low-latency inference using SRAM and in-memory compute, not on throughput. Customers now deploy racks and clusters, so D-Matrix moved from chip scale to rack scale. Raptor stacks custom DRAM on compute. Sid says this gives roughly 20–30x more capacity per rack. Timestamps 0:00 – Intro: from training to low-latency inference 0:43 – Why start a chip company in 2019, before ChatGPT 4:04 – Betting on inference for the data center and cloud 7:08 – Transformers, GPT-3 and finding product-market fit 9:16 – The first chip and the in-memory compute demo 14:25 – How a first chip gets built: partners, TSMC and COVID 17:10 – From a digital in-memory engine to a first product in 2024 20:24 – What "rack scale" means and why the industry moved to it 24:01 – D-Matrix vs. GPUs: throughput vs. interactivity, and no HBM 26:58 – The business model and target customers 29:07 – The Nvidia partnership and the stacked-DRAM product, Raptor 34:32 – The two acquisitions and how they fit 37:06 – The AI hardware investment cycle 40:05 – What success looks like for D-Matrix

d-Matrix Startup Project: Build the future GPT-3 Raptor Claude

Chapters

  1. 0:00 Intro: from training to low-latency inference
  2. 0:43 Why start a chip company in 2019, before ChatGPT
  3. 4:04 Betting on inference for the data center and cloud
  4. 7:08 Transformers, GPT-3 and finding product-market fit
  5. 9:16 The first chip and the in-memory compute demo
  6. 14:25 How a first chip gets built: partners, TSMC and COVID
  7. 17:10 From a digital in-memory engine to a first product in 2024
  8. 20:24 What "rack scale" means and why the industry moved to it
  9. 24:01 D-Matrix vs. GPUs: throughput vs. interactivity, and no HBM
  10. 26:58 The business model and target customers
  11. 29:07 The Nvidia partnership and the stacked-DRAM product, Raptor
  12. 34:32 The two acquisitions and how they fit
  13. 37:06 The AI hardware investment cycle
  14. 40:05 What success looks like for D-Matrix