Why GPUs Aren't the Answer for Every AI Workload | Sid Sheth Founder and CEO of d-Matrix
People
-
Nataraj Host
-
Sid Sheth Guest
Listen on
Sid Sheth, co-founder and CEO of D-Matrix, joins Nataraj on Startup Project to talk about the future of AI inference, why GPUs may not be the answer for every workload, and what it takes to build a semiconductor startup.Key takeaways Training and inference are like education and a career. Sid's view is that inference is the much larger opportunity. D-Matrix focuses on ultra-low-latency inference using SRAM and in-memory compute, not on throughput. Customers now deploy racks and clusters, so D-Matrix moved from chip scale to rack scale. Raptor stacks custom DRAM on compute. Sid says this gives roughly 20–30x more capacity per rack. Timestamps 0:00 – Intro: from training to low-latency inference 0:43 – Why start a chip company in 2019, before ChatGPT 4:04 – Betting on inference for the data center and cloud 7:08 – Transformers, GPT-3 and finding product-market fit 9:16 – The first chip and the in-memory compute demo 14:25 – How a first chip gets built: partners, TSMC and COVID 17:10 – From a digital in-memory engine to a first product in 2024 20:24 – What "rack scale" means and why the industry moved to it 24:01 – D-Matrix vs. GPUs: throughput vs. interactivity, and no HBM 26:58 – The business model and target customers 29:07 – The Nvidia partnership and the stacked-DRAM product, Raptor 34:32 – The two acquisitions and how they fit 37:06 – The AI hardware investment cycle 40:05 – What success looks like for D-Matrix
Chapters
- 0:00 Intro: from training to low-latency inference
- 0:43 Why start a chip company in 2019, before ChatGPT
- 4:04 Betting on inference for the data center and cloud
- 7:08 Transformers, GPT-3 and finding product-market fit
- 9:16 The first chip and the in-memory compute demo
- 14:25 How a first chip gets built: partners, TSMC and COVID
- 17:10 From a digital in-memory engine to a first product in 2024
- 20:24 What "rack scale" means and why the industry moved to it
- 24:01 D-Matrix vs. GPUs: throughput vs. interactivity, and no HBM
- 26:58 The business model and target customers
- 29:07 The Nvidia partnership and the stacked-DRAM product, Raptor
- 34:32 The two acquisitions and how they fit
- 37:06 The AI hardware investment cycle
- 40:05 What success looks like for D-Matrix