Walter Goodwin on Fractile's 3D-stacked memory chips, AI infrastructure bottlenecks, and NVIDIA's future valuation
Summary
Walter Goodwin, CEO of Fractile, argues that frontier AI intelligence is not yet commoditized, distinguishing it from general cloud compute. He positions Fractile's 3D-stacked memory architecture as a solution to the memory bandwidth bottleneck that limits LLM inference speeds. Goodwin contends that land and power, rather than silicon, are now the primary constraints for AI infrastructure, favoring energy-efficient designs. He predicts NVIDIA will reach a $10 trillion valuation within five years, driven by the finite nature of data center power and the need for chip replacement cycles.
Episode sections
- 0:00 – 4:17
Intro and Fractile Overview
Speaker 0 discusses the massive future compute needs for AI, noting that frontier intelligence is not yet commoditized. Speaker 1 introduces Walter Goodwin, CEO of Fractile, highlighting the company's $720M Series C raising at a $7.5B valuation led by Thrive, Founders Fund, Lightspeed, and Redpoint.
- 4:17 – 7:12
Chip Company vs. Vertical Integration
Goodwin defines Fractile as a chip company that also employs ML researchers to understand model evolution, though it monetizes only by selling chips and racks, not internal models. He argues that while vertical integration is a trend among labs to avoid commoditization, Fractile competes at the chip layer without conflicting with customers.
- 7:12 – 12:05
Commoditization and Neo Clouds
Goodwin distinguishes between commoditized cloud compute and non-commoditized frontier intelligence, citing DeepSeek's ability to maintain high margins through a full-stack inference optimization. He asserts that neo-clouds are vertically integrating to escape the commoditization of their compute layer.
- 12:05 – 16:15
Specialist Chips vs. NVIDIA
Goodwin positions Fractile and other specialist chips as contrarian bets offering 10x advantages in cost and speed for specific workloads, contrasting with NVIDIA's general-purpose, high-volume approach. He notes that while NVIDIA excels at inference at scale, the frontier requires specialized bets on evolving architectures.
- 16:15 – 20:34
The Memory Wall and HBM Limitations
Goodwin explains that LLM inference is bottlenecked by memory bandwidth rather than compute power, as scaling memory bandwidth has lagged significantly behind flop count scaling. He describes the physical limitations of DRAM access via copper wires and the trade-off between SRAM's speed and DRAM's density.
- 20:34 – 27:37
3D Stacking and Fractile's Architecture
Fractile is developing 3D stacking technology to place memory directly on top of the logic die, aiming to combine SRAM-like bandwidth with DRAM-like capacity. Goodwin contrasts this with HBM-based chips from competitors like NVIDIA, OpenAI, and Etched, which are limited by the same off-chip memory modules.
- 27:37 – 29:59
Production Status and Manufacturing
Goodwin confirms Fractile has silicon that has gone through foundries and is running trials, with full product tapeouts expected in a few months. He highlights Fractile's in-house backend design team as a strategic advantage over competitors who rely on intermediaries like Broadcom to access TSMC and HBM capacity.
- 29:59 – 30:55
Broadcom's Role and Market Structure
Goodwin explains that many major chip projects (TPU, MTIA, Jalapeno, etc.) share the same underlying Broadcom-designed platform, differing only in naming and minor optimizations. He suggests this reveals a market structure where many distinct chips are built on identical foundational stacks.
- 30:55 – 35:49
Post-Tapeout Complexity and 'Task Zero'
Speaker_0 explains that tapeout is merely the handoff to the foundry, not the completion of the product. The critical milestone is 'task zero,' defined as the day a reference workload runs on a full rack of chips. Success depends on manufacturability and yield, not just silicon design.
- 35:49 – 39:03
Shifting Bottlenecks: From Silicon to Land and Power
Speaker_0 argues that while silicon and memory shortages were once the primary constraints, land and power are now the fundamental bottlenecks for AI infrastructure. This scarcity incentivizes the development of chips that are highly efficient per megawatt, rather than just fast.
- 39:03 – 42:04
Energy Supply Constraints and Geopolitics
Speaker_0 highlights a massive energy deficit, noting grid connection backlogs in the US and UK are orders of magnitude larger than current grid capacity. China is described as having an advantage due to its ability to rapidly deploy large-scale energy projects, such as hydro, near compute sites.
- 42:04 – 50:10
Inference as the Primary Value Driver
Speaker_0 asserts that training is a fixed cost while inference is the variable cost that matters for ROI. With the advent of 'test-time compute' and reasoning models, inference demand is scaling exponentially, requiring massive amounts of compute to solve complex problems.
- 50:10 – 53:58
Token Cost and Value Accrual
Speaker_0 predicts that while token costs will drop, total spending on inference will rise significantly over the next decade. He believes value will accrue primarily to chip manufacturers and frontier model labs, as software orchestration layers may struggle to sustain margins amid hardware heterogeneity.
- 53:58 – 1:15:33
Critique of Chip Heterogeneity and Orchestration
Speaker_0 argues that mixing different chip types (e.g., GPUs and SRAM-based chips) leads to stranded capacity and inefficient resource allocation. He contends that integrated solutions, like Fractile's stacked memory architecture, offer a superior path by combining compute, bandwidth, and capacity in a single unit.
Assets discussed
| Asset | Kind | Stance | Context |
|---|---|---|---|
| NVIDIANVDA | equity | neutral | What does that do to the NVIDIA business? If we think about kind of NVIDIA as a horizontal provider and you as a kind of specialized provider, is the future of chips specialized according to workloads? |
| OpenAI | private company | neutral | Whether it's you know the GPUs, TPUs, the Jalapeno chip from OpenAI, etched, MatX, they have often really brilliant kind of architectures on the chip |
| Meta PlatformsMETA | equity | neutral | whether it's the TPU or jalapeno, MTIA from Meta, MatX |
“Stance” is how the speakers talked about the asset, not a recommendation.
Claims worth verifying
Fractile raised $720M in Series C, valuing the company at $7.5B.
Unidentified speaker (UNKNOWN) ·0:28
How to verify: Fractile press release or SEC Form D filing for the Series C round. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Primary sourceThrive, Founders Fund, Lightspeed, and Redpoint led Fractile's Series C.
Unidentified speaker (UNKNOWN) ·0:28
How to verify: Fractile press release or SEC Form D filing listing lead investors. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Primary sourceFrontier intelligence is not yet commoditizing.
Unidentified speaker (speaker_0) ·0:00
How to verify: Market analysis of pricing trends for frontier vs. open-source LLM inference services. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
DeepSeek released its entire inference stack, achieving higher throughputs than competitors on GPUs.
Unidentified speaker (speaker_0) ·9:49
How to verify: DeepSeek GitHub repository or technical blog post detailing the inference stack performance benchmarks. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Memory bandwidth has scaled only 40-fold in 25 years, while flop counts scaled a million-fold.
Unidentified speaker (UNKNOWN) ·16:22
How to verify: Historical semiconductor industry reports (e.g., from IEEE or semiconductor trade journals) comparing GPU flop rates and memory bandwidth over the last 25 years. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Fractile has silicon that has gone through foundries and is running trials, with full product tapeouts in a few months.
Unidentified speaker (UNKNOWN) ·28:00
How to verify: Fractile press releases or announcements regarding tapeout milestones and production status. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Broadcom provides identical underlying platforms to TPU, MTIA, Jalapeno, and Etched.
Unidentified speaker (speaker_0) ·30:23
How to verify: Industry analysis reports or supply chain disclosures confirming Broadcom's role as a backend partner for these chip designs. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Primary sourceNVIDIA builds the best AI chips in the world by many metrics.
Unidentified speaker (UNKNOWN) ·13:43
How to verify: Independent benchmark comparisons (e.g., MLPerf) comparing NVIDIA GPUs against other AI accelerators. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Primary sourceLLM inference speed is bottlenecked by memory bandwidth, not compute power.
Unidentified speaker (speaker_0) ·18:04
How to verify: Technical whitepapers or performance analysis from chip architects explaining the memory wall in LLM inference. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Fractile is developing 3D stacking technology to place memory on top of logic dies.
Unidentified speaker (speaker_0) ·21:34
How to verify: Fractile technical publications, patents, or press releases detailing their 3D integration approach. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Fractile has raised $940M, including a $720M Series C valued at $7.5BN.
Unidentified speaker (speaker_0) ·30:55
How to verify: Check Fractile's investor relations page or press releases for the Series C announcement and total funding raised. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Tapeout is the moment the GDS2 file is sent to the foundry to build mask sets.
Unidentified speaker (speaker_0) ·32:17
How to verify: Verify against standard semiconductor industry definitions of tapeout and GDS2 files. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Grid connection backlogs in the US and UK are an order of magnitude larger than current grid energy capacity.
Unidentified speaker (UNKNOWN) ·39:08
How to verify: Check EIA (Energy Information Administration) or UK National Grid reports on connection queues and capacity. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
NVIDIA acquired Grok.
Unidentified speaker (speaker_0) ·54:51
How to verify: Check SEC filings for NVIDIA's acquisition of Grok. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Primary sourceNVIDIA and Grok will deploy Ruben GPUs and Grok chips together for pre-fill and decode disaggregation.
Unidentified speaker (speaker_0) ·54:51
How to verify: Review NVIDIA's technical presentations or press releases on Grok integration and inference architectures. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Primary sourceOpenAI signed a deal with Cerebrus.
Unidentified speaker (speaker_0) ·59:48
How to verify: Check public announcements or press releases from OpenAI and Cerebrus regarding their partnership. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Both NVIDIA's Grok acquisition and OpenAI's Cerebrus deal were signed on December 24th.
Unidentified speaker (speaker_0) ·59:48
How to verify: Check the exact dates of the press releases or SEC filings for these transactions. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Primary sourceJensen Huang praised Fractile in London.
Unidentified speaker (speaker_0) ·59:08
How to verify: Search for news articles or transcripts from Jensen Huang's visit to London where he discussed Fractile. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Microsoft pulled out of the NScale project.
Unidentified speaker (UNKNOWN) ·41:37
How to verify: Check news reports or press releases from Microsoft and NScale regarding the project's cancellation. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Primary sourceFractile raised a $720 million Series C.
Unidentified speaker (speaker_0) ·1:02:39
How to verify: Fractile's investor relations page or press releases from 2026-08-14 through 2026-11-14; SEC Form D if filed. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Primary sourceFractile was valued at $7.5 billion in the Series C.
Unidentified speaker (speaker_0) ·1:02:39
How to verify: Fractile press releases or investor updates from 2026-08-14 through 2026-11-14. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Dan Rabinovich from Meta joined Fractile a few weeks before the interview.
Unidentified speaker (speaker_0) ·1:05:49
How to verify: LinkedIn profile of Dan Rabinovich or Fractile's official hiring announcements from 2026-08-14 through 2026-11-14. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
TSMC underpins all of the world's chip production, particularly frontier chips.
Unidentified speaker (UNKNOWN) ·1:06:36
How to verify: TSMC annual reports or market share data from semi-conductor industry analysts (e.g., Gartner, IDC) for 2026. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Inherent AI is a startup in London focused on automating science and research.
Unidentified speaker (speaker_0) ·1:07:19
How to verify: Inherent AI website or press coverage from 2026-08-14 through 2026-11-14. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Verifiable math and coding challenges will be solved by frontier models within the next twelve months.
Unidentified speaker (speaker_0) ·1:08:56
How to verify: Tracking AI model benchmarks (e.g., SOTA on math/coding leaderboards) over the 12 months following the interview date. For recent events, verify 2026-08-14 through 2026-11-14. Use another period only when the speaker explicitly names it.
Do your own research
- Verify Fractile's 'Task Zero' status by checking engineering blogs or press releases for rack-level inference announcements.
- Review EIA data on US/UK grid connection backlogs and data center permitting timelines to assess land/power constraints.
- Analyze SEC filings for NVIDIA's acquisition of Grok and public announcements regarding OpenAI's partnerships.
- Check Fractile's technical whitepapers or patent filings for details on their 3D stacking process.
- Monitor revenue growth rates and margins of chip companies vs. AI software platforms to test value accrual claims.
- Compare energy permitting timelines and renewable energy project approvals in China vs. the US/UK.
- Evaluate CapEx amortization models for data centers deploying mixed-architecture fleets to assess stranded capacity risks.
- Review research papers from OpenAI and other labs on 'reasoning models' and 'chain-of-thought' compute scaling.
Full transcriptexpand
- 0:00
- Unidentified speaker (speaker_0): We're so traumatized by the dot com bubble that we've priced in exuberance right from the outset. We will be spending far more on running models in 10 years' time than we are today. It feels like we need thousands of times more compute than we have today. Frontier intelligence is not yet commoditizing. I think
- 0:19
- Unidentified speaker (speaker_0): we'll have a couple of multi-hundred Multi hundred billion dollar companies built in that space. One neo cloud that is competent is much like another neo cloud. that is competent.
- 0:28
- Unidentified speaker (UNKNOWN): This
- 0:28
- Unidentified speaker (speaker_1): is 20 VC with me, Harry Stabbings. Now joining me in the hot seat today is one of the leading minds when it comes to Frontier chips. Walter Goodwin, founder and CEO of Fractile. The business has just raised a $720 million Series C, co-led by Thrive, Founders Fund, Lightspeed, Light speed and Radpoint, red point, valuing the company at a seven $7
- 0:48
- Unidentified speaker (speaker_1): billion dollar price. As I said, Walter is one of the leading minds in the future of chips, and we get into that and so much more in the discussion today. But before we dive into the show today, founders and investors sleep on 8 sleep. It is the intelligent sleep system trusted by Mr. Mark Zuckerberg, Dr. Andrew Huberman, Charles LeClerc,
- 1:08
- Unidentified speaker (speaker_1): and the Aston Martin F1 team. The pod is a smart mattress cover that fits on any bed. It cools each side. to 55 degrees, heats to 110 degrees. It even tracks your heart rate, HRV, and sleep stages without a wearable. Thank God. The new pod 6 has a hub 50% smaller than the last one and
- 1:28
- Unidentified speaker (speaker_1): starts at just $1,999. Visit 8Sleep.com slash 20 V C. That's E-I-G-H-T-S-L-E-E-P E I G H T S L E P dot com. slash 20 VC. While 8Sleep helps you sleep better, MongoDB helps you build better. MongoDB has always been the
- 1:47
- Unidentified speaker (speaker_1): database developers love. Well, now it's the data platform AI agents need. Agents need accurate context, fast. MongoDB stores, searches, and reasons over your data in real time. JSON native database, Vector Search. and Voyage AI embeddings all in one place. One system instead of 10, no
- 2:07
- Unidentified speaker (speaker_1): data pipelines to maintain. Ugh, this sounds too good to be true. Build and scale from your first user to billions of vectors, run on any cloud, on prem, or your laptop even. That's why 75% of the Fortune 100 run their most critical apps on MongoDB, moving trillions of dollars every single day. And it's why AI. native
- 2:27
- Unidentified speaker (speaker_1): companies like Eleven Labs run 40 million agents on MongoDB. So if you're building an AI, MongoDB for startups helps you move faster with Atlas and Voyage AI credits and dedicated support. Don't build agents that answer once and forget. Build agents that remember and learn from your real-time Real time data. Go to
- 2:46
- Unidentified speaker (speaker_1): MongoDB.comslash mongoDb.com slash agents. That's M-O-N-G-B. m-o-n-g-odb.com slash agents. While MongoDB scales the product, Framer scales the story. So if your team wants a website that looks and feels handcrafted, but is really still fast to ship, Framer is built for exactly that. Framer
- 3:06
- Unidentified speaker (speaker_1): is the AI website builder that helps creators, teams, and businesses ship production ready sites faster than ever while getting every detail right. Prompt in inspect, edit, and publish in one place at a whole new pace. Agents and humans work in tandem. Agents bring speed and scale. You bring taste, judgment and control. The
- 3:25
- Unidentified speaker (speaker_1): work lands on the canvas and stays editable. Build custom code components, manage CMS content, optimize SEO, and audit for issues all in one place. Enterprise grade hosting, security and 99.9%. 99.99% uptime SLAs, trusted by leading brands like Perplexity and Miro. Learn how you can get more out of
- 3:45
- Unidentified speaker (speaker_1): your site from a Framer specialist or get started building for free today at Framer.com slash 20VC for 30% off a Framer Pro annual plan. That's Framer.com slash 20VC. 20 VC, Rules and restrictions may apply.
- 4:00
- Unidentified speaker (speaker_2): You have now arrived at your destination.
- 4:03
- Unidentified speaker (speaker_3): Walter, I am so
- 4:04
- Unidentified speaker (speaker_1): excited for this dude. It's lovely to be able to do it in person. And I'm so thrilled that you could join me in the studio, ma'am.
- 4:10
- Unidentified speaker (speaker_0): Very good to be here with you, Harry.
- 4:12
- Unidentified speaker (speaker_1): Now I want to start. For those that don't know, 60 seconds, like what is fractal? What do we need to know?
- 4:17
- Unidentified speaker (UNKNOWN): We're
- 4:17
- Unidentified speaker (speaker_0): a Frontier AI company of sorts. But we make chips, we make these big rack scale systems that are, you know, fully focused on advancing the frontier of AI capabilities. And so from a team composition point of view, this means that we have ML researchers down to process experts that fly out to Taiwan
- 4:37
- Unidentified speaker (speaker_0): every few weeks, down to people that actually look at the thermals and the mechanicals of putting together one of these rack scale systems.
- 4:44
- Unidentified speaker (speaker_1): I was listening to you on No Prize, we were just talking about Sarah. And you were saying unequivocally that you were a chip company. And I was listening to this thinking, great. But then I'm looking at it going, everyone's moving full stack. No, we're seeing OpenAI have jalapeno, we're seeing DeepSeek have their own, we're seeing Anthropic build their own, obviously Meta and Mark has his own. My question is, is
- 5:04
- Unidentified speaker (speaker_1): the future won by vertical integration? And can you be a chip company alone these days?
- 5:09
- Unidentified speaker (UNKNOWN): It's
- 5:09
- Unidentified speaker (speaker_0): a very good question. I think there's a verticalization in that as all of these different components.
- 5:15
- Unidentified speaker (speaker_0): come together, and especially as you kind of look into the future and you work out, you know, what are these models going to look like? How do the workloads evolve? How does the silicon evolve? There is this absolute necessity to kind of have a vision on every part of that. So Fractile is a chip company, but in a sense, we're also a Frontier AI company. And I think what that means for us is, you know, and this will continue to be the
- 5:35
- Unidentified speaker (speaker_0): case, you know, we monetize by selling chips, we monetize by selling racks of compute to our customers. But it's completely essential that we have a bunch of folks internally who are ML researchers, who are looking at how model architectures might evolve, who are maybe even running smaller scale experiments than the ones that you would see in the leading labs.
- 5:55
- Unidentified speaker (speaker_0): And so I think there is a distinction here between the expertise that is needed internally to win and then the kind of nature of the business, what the business model is. You know, we we never intend to compete with our customers. We will never monetize internal models because that's how we've chosen to build this company.
- 6:15
- Unidentified speaker (speaker_0): We've chosen to compete in the chip layer, but you can't sort of step back from challenging yourself to have expertise in all of these other places, because you know, in our case, that would mean you'd place the wrong bets. You have to synthesize everything that you're learning across the ecosystem, and you have to bring that back then to be
- 6:35
- Unidentified speaker (speaker_0): obsessive about the part of this space that you're building. And if you do this right, you've ideally found yourself an enormous sort of piece of green space where you're incredibly helpful to your partners, to your customers, and to your suppliers upstream. And you've kind of got to know part. parts of their business as well. But of course, you never seek to conflict with them. Aaron
- 6:55
- Unidentified speaker (UNKNOWN): Powell
- 6:55
- Unidentified speaker (speaker_1): Would you argue that everyone's moving to vertical integration because everyone's fighting against commoditization of one layer? If you think about what OpenAI did with Jalapeno in nine months, it suggests some form of commoditization of the layer. Do you think everyone is desperately expanding to run against commoditization?
- 7:12
- Unidentified speaker (UNKNOWN): I
- 7:12
- Unidentified speaker (speaker_0): think there's definitely the fight to keep your margins expanded that do mean that, let's say if you are serving frontier models, you know, it is important that you're not squeezed all the way down by very high margins on incumbent chip players. And that's where it's very useful to have, for example, a first-party play or another third-party
- 7:32
- Unidentified speaker (speaker_0): play that can It be a counterpoint to that. And so I think, in a sense, there is this confluence of forces that do mean that, you know, if you're a Frontier lab, if you're a hyperscaler, it's very important to kind of have your own retort to various parts of this. I do also think that there's this pretty crucial distinction today still between what
- 7:52
- Unidentified speaker (speaker_0): is perhaps becoming commoditized and you know what we still see sort of at the frontier.
- 7:58
- Unidentified speaker (speaker_1): What is commoditized then? Help me. I'm an ambassador. investor, Okay. I I I look for value in amazing entrepreneurs and businesses. What is commoditized that most people think isn't and what isn't that maybe people think is?
- 8:09
- Unidentified speaker (speaker_0): That's a great question. There is this sort of sometimes concern, sometimes excitement about the idea of intelligence saturation, you know, that as models get smarter and smarter, there will become a kind of waterline that they exceed where we don't really need greater intelligence for a whole set of tasks. And I think it is true that there are certain tasks for
- 8:29
- Unidentified speaker (speaker_0): which we see this kind of intelligence saturation. And this is where you then see, you know, an open model is good enough. And these models become commoditized at that layer. I think where that becomes more nuanced, though, and where actually models, even at that level, sort of still fight against commoditization, is that there is always
- 8:49
- Unidentified speaker (speaker_0): economic pressure to for example, reduce the cost of intelligence. And perhaps counterintuitively, you know, this is actually why I think the frontier labs and the frontier efforts remain so vital, is of course, frontier intelligence is not yet commoditizing. We see large margins there, we see big deltas against, you know,
- 9:09
- Unidentified speaker (speaker_0): against sort of everybody else. And also these sort of learnings from the frontier, they kind of trickle down. also into building better, you know, better, smaller models as well. And so actually, you know, if you look at that sort of fight across the entire gamut of frontier LLM capability as well as kind of behind the frontier LLM capability,
- 9:29
- Unidentified speaker (speaker_0): there's still this sort of ferocious battle. And I think, you know, I would distinguish I or I would characterize that as as sort of not commoditized insofar as if you are a market leader in something, you can still extract high margins. So a good example of this actually is uh DeepSeek released a few days ago their entire like inference stack, their entire
- 9:49
- Unidentified speaker (speaker_0): software stack. And it had really remarkable you know, people very quickly characterized it and realized that DeepSeek had done so well in building an inference stack to run their models on GPUs that they were getting much higher throughputs, much better. interactivity, better speeds than everybody else who had built a DeepSeek inference server. And so something that you
- 10:09
- Unidentified speaker (speaker_0): might have thought would have been by this point commoditized, which is the serving of an open model. In their case, by going full stack, they'd built a model which was more efficient, and they built an inference server for that model that was more efficient. And they'd carved themselves out the right to have really healthy margins on that. So I think even in the sort of open source space, we
- 10:29
- Unidentified speaker (speaker_0): don't yet see this kind of complete race to the bottom, at least for those folks that sort of again try to own a lot of vertical understanding of the space. I think what is commoditizing, and I don't think anybody has ever argued it won't be, is kind of, you know, the whole space of cloud compute. You know, one neo cloud that is competent is much
- 10:49
- Unidentified speaker (speaker_0): like another neo cloud that is competent. The reason it's so exciting today is we don't yet know which ones are competent. So I think there's still a lot of space there for there to be huge winners and maybe sort of huge losers as well.
- 11:01
- Unidentified speaker (speaker_1): But I think that's why you see all the neo clouds desperately trying to vertically integrate two, just because I think they're very aware of the commoditization of their own layer. You know, we had the founder of Nebius on the show who was talking about the movement into model routing and inference for them as a core business. and a core strategic, just because I think they're aware of that. I
- 11:18
- Unidentified speaker (UNKNOWN): think
- 11:18
- Unidentified speaker (speaker_0): that's true. And you definitely see, you know, every neo cloud is trying to build an inference server. A lot of these folks who built an advantage in inference serving, you know, which is a software advantage ultimately are also thinking about how they can have more control of their destiny on the compute side. Obviously, I'm biased, but I think this is where the kind of neo chip story and the you know
- 11:38
- Unidentified speaker (speaker_0): differentiated the
- 11:39
- Unidentified speaker (speaker_0): silicon can actually become a really core differentiator for those players as well.
- 11:44
- Unidentified speaker (speaker_1): Can you unpack that? What is that?
- 11:45
- Unidentified speaker (speaker_0): So we're still in the very early innings of the kind of rollout of silicon that is not the mainstream conventional, you know, GPUs and TPUs and other XPUs. And it is still much bolder, especially if you look at the kind of way that these build build-outs outs get financed, to lean in into a new chip platform, a
- 12:05
- Unidentified speaker (speaker_0): fractile, a cerebrus, and so on.
- 12:07
- Unidentified speaker (speaker_1): And we need these new sorry just to bring everyone along with us. We need these new chip platforms because workloads are becoming more and more complex and intense. And the specialization of these chips will allow for the execution of these workloads to be much more efficient. Aaron
- 12:19
- Unidentified speaker (UNKNOWN): Powell
- 12:19
- Unidentified speaker (speaker_0): I think these new platforms, you know, obviously we're one of these, you know, have a belief that in certain workloads you have the this kind of more than 10x advantage, whether it's on cost or in our case, it's cost and speed. And so, you know, in a sense, they are contrarian bets still. You know, we still don't see many gigawatts of new silicon built out,
- 12:39
- Unidentified speaker (speaker_0): but they do kind of have this potential to fundamentally alter the economics. And so if you imagine you are a neo cloud and you're the first neo NeoCloud cloud to work out how to break out of a particular bond with a particular GPS. GPU vendor, potentially you get to reap this enormous benefit. And we've seen this with uh, you know, Fluid Stack is a great example, right? They were in some ways the first
- 12:59
- Unidentified speaker (speaker_0): NeoCloud to kind of decide that they were going to step bodily away from NVIDIA. And they deployed a bunch of Google TPUs. And they've had an extraordinary year as they sort of cross that Rubicon and earn the right to. start deploying other chips. And so I think there are opportunities to,
- 13:20
- Unidentified speaker (speaker_0): as you kind of go more vertical, also take some of these sort of slightly higher risk bets, I guess, and reap huge benefits from that. Aaron
- 13:28
- Unidentified speaker (UNKNOWN): Powell
- 13:28
- Unidentified speaker (speaker_1): What does that do to the NVIDIA business? If we think about kind of NVIDIA as a horizontal provider and you as a kind of specialized provider, is the future of chips specialized according to workloads? And how do we think about that market composition horizontal versus specialist? Aaron
- 13:43
- Unidentified speaker (UNKNOWN): Powell
- 13:43
- Unidentified speaker (speaker_0): The way that I think about this, you know, NVIDIA is an extraordinary business and they build by many metrics, in a sense, the best AI chips in the world, right? And the best systems, and they have extraordinary work on data center scale inference, for example. So in the inference space, I think they've done some some brilliant work as well. What I think is different
- 14:03
- Unidentified speaker (speaker_0): about this moment in time and and and sort of caught a fractile's identity is there is this idea of kind of an evolving frontier. And you know, the frontier necessitates slightly different bets and a slightly different way of thinking because you are necessarily setting up a business that is able to kind of look
- 14:23
- Unidentified speaker (speaker_0): around corners, is able to make bold predictions. and is then able to steer the entire tech product, for example, towards those bold predictions. And I think if you look at NVIDIA's business, what they excel at is delivering high-quality products at extraordinary volumes that are, in a sense,
- 14:43
- Unidentified speaker (speaker_0): unopinionated, right? These are powerful general purpose processes that are very good at various parts of, for example. you know, the AI compute world. If you look at where Fractile's bet is on inference, for example, which is building chips that are orders of magnitude faster than these GPUs, that's a particular focus. And for us architecturally,
- 15:03
- Unidentified speaker (speaker_0): it's meant this focus on finding paths to huge memory bandwidth, for example. And it's potentially taken us away in some ways from some of that complete generality. I think what's interesting though today, and the reason, of course. that we're so excited about this space is the frontier is kind of also an overwhelming portion of the total market. You know, if you look at
- 15:23
- Unidentified speaker (speaker_0): what's happened just over the last few years, we have this enormous build out of specifically inference compute to serve LLMs at the frontier. And so, you know, what is in some sense specialist is also by no means a new niche. And I think that's this sort of real transformation that if you look back, you know, 10 years,
- 15:43
- Unidentified speaker (speaker_0): even five years, it would have been slightly crazy to build a kind of specialist chip, right? We didn't have any single workload that made up sort of more than, you know, let's say 10% of the total compute burden. Suddenly we do. And I think that demands slightly different company structures that can have that kind of predictive power built
- 16:03
- Unidentified speaker (speaker_0): in that try to also orchestrate the entire org around what it means to chase after that.
- 16:09
- Unidentified speaker (speaker_1): You mentioned memory and HBM there. Why is the memory wall such a prominent problem still today? Aaron
- 16:15
- Unidentified speaker (UNKNOWN): Powell
- 16:15
- Unidentified speaker (speaker_1): Yeah,
- 16:15
- Unidentified speaker (speaker_0): it's a it's one of these sort of interesting facts about kind of chip physics. If you look at the two
- 16:22
- Unidentified speaker (UNKNOWN): great
- 16:22
- Unidentified speaker (speaker_1): days start with that line.
- 16:25
- Unidentified speaker (speaker_2): So
- 16:25
- Unidentified speaker (speaker_0): if you look at if you look across the last like twenty years in in the silicon world, there's sort of two things that really make up the kind of top level specs, the kind of top trumps of your chip, right? And like one of them, certainly for AI chips, is the flop count. How many floating point operations can you do per second? This is like a multiply or an add. And so that's your kind of mathematical power on
- 16:45
- Unidentified speaker (speaker_0): that chip. And then the other piece is is kind of the properties of the memory on that chip, the capacity and in particular the bandwidth to that memory. And that's how quickly you can shuttle data back and forth from you're looking at a GPU, you actually have that silicon in the middle of it that is running the maths. It's got that high flop count. And then there's other pieces of silicon around
- 17:05
- Unidentified speaker (speaker_0): it, the HBM, that are kind of coming in on this on this sort of highway, bringing data in. And we've been extraordinarily good with Moore's law at scaling flop count. We've scaled it about a million fold in the last 25 years on these chips. Bandwidth, by contrast, is kind of much more intrinsically physics limited. We've scaled it about
- 17:24
- Unidentified speaker (speaker_0): 40 fold in the same time. So it's lagged behind the flop scaling about 25,000 times. And so we've naturally found ourselves more and more kind of bandwidth limited in a lot of the operations that we do. And for something like LLM inference, this has become a kind of critical bottleneck because of this
- 17:44
- Unidentified speaker (speaker_0): interplay between the generation of text, which I think we all sort of have this sense that it's this sort of auto-aggressive, you know, sequential thing, one word after another is generated out of these models. Every time I go and generate one of those words, I'm loading from that memory over that limited bandwidth. all of the weights of my model.
- 18:04
- Unidentified speaker (speaker_0): And today also all of the context that I'm operating on as well. So all of the path states of things that I've seen. And it's that that is then my bottleneck on generation speed. You know, the reason that if you take a very large frontier model, it might generate at 80 tokens per second, 80 words per second, is going to be not because of the limited maths capability on the
- 18:24
- Unidentified speaker (speaker_0): chip, but because you don't have nearly enough. bandwidth to that memory. I cannot load the weights of the model from memory into that processor more than, let's say, 80 times per second. And the reason I started with, you know, the sort of sexy line about physics here is it is if you zoom in at the interface, there's a bunch of essentially
- 18:44
- Unidentified speaker (speaker_0): copper wires that are going from this DRAM memory into the processor. And we've been pretty good, the world's been pretty good at making them. smaller and smaller and smaller. But there's only sort of so much you can scale that. And so you have, let's say, a few thousand wires. Aaron
- 18:57
- Unidentified speaker (UNKNOWN): Powell
- 18:57
- Unidentified speaker (speaker_1): And is that why, sorry, you have SRAM? Am I right
- 19:00
- Unidentified speaker (UNKNOWN): in
- 19:00
- Unidentified speaker (speaker_1): saying that? Moving memory closer to it. Aaron
- 19:02
- Unidentified speaker (UNKNOWN): Ross Powell
- 19:03
- Unidentified speaker (speaker_1): So
- 19:03
- Unidentified speaker (speaker_0): SRAM, yeah. So there's this sort of, you know, DRAM is like off in a different building, basically. And you're carrying these boxes down the stairs and then up the stairs again into the into the processor building where the maths is done. And you have to carry them back. again, unfortunately, because there's not enough space in that building, let's say, to to store all the boxes. So actually, maybe it's a warehouse and then you've got the office where you're doing
- 19:23
- Unidentified speaker (speaker_0): you're doing the work. SRAM is like having a bunch of boxes that are permanently in the processor, right? So that's a small amount of memory that you cram onto the same silicon where way you're doing your processing. But that's the that's the snag as well. The reason that over the last 30 years we've externalized memory into this this thing called DRAM
- 19:43
- Unidentified speaker (speaker_0): and put it on different silicon, these are built in completely different foundries. They've been specialized for very, very different things. So DRAM is completely focused on density. The DRAM bit cell as it's called is an incredibly simple structure and you can cram you know many billions of these into a very small area of silicon. SRAM by
- 20:03
- Unidentified speaker (speaker_0): contrast, is built conveniently on the same exact processed nodes. You know, it's built on the silicon that you're building your processor by TSMC, for example, but it isn't optimized for density. So you might have less than 10 times as much kind of SRAM capacity per unit area as you can get sort of gigabytes of DRAM on that same
- 20:23
- Unidentified speaker (speaker_0): area. And by extension per dollar as well, because your silicon wafer costs you. a certain amount. Aaron
- 20:27
- Unidentified speaker (UNKNOWN): Ross
- 20:27
- Unidentified speaker (speaker_0): Powell
- 20:28
- Unidentified speaker (speaker_1): When you look at inference and the future of inference, which is then best suited? Is it DRAM or SRAM? Because latency is important. Aaron
- 20:34
- Unidentified speaker (UNKNOWN): Powell
- 20:34
- Unidentified speaker (speaker_0): That's right. So latency and and bandwidth as well. Like how much can I just cram through this? So this is one of the things that we've been working on over the past couple of years at Fractile, is I guess the rejection of this trade-off. So SRAM as a memory is the only one today that can be built on the same processor. It's the only one that can be incredibly close and have incredibly high
- 20:54
- Unidentified speaker (speaker_0): bandwidth. DRAM, much cheaper, much denser, much more capacity, but fundamentally limited in the way that we access it. But if you look at that that sort of physics, the fact that it's coming in over wires, it's because we've kind of got it in the wrong place, right? We've got the DRAM sort of sitting in plane with the chip, and we've got this like limited, it's called a beach.
- 21:14
- Unidentified speaker (speaker_0): beachfront, right? It's like literally the limited kind of real estate on the edge of the chip. What we've been working on at Fractile over the past couple of years is essentially stacking that memory on top of the logic. So looking for ways to rather than have this kind of limited beachfront, you know, everybody gets to have a a seaside property, basically. It's
- 21:34
- Unidentified speaker (speaker_0): like the entire memory layer sits on top of that logic die. Aaron
- 21:38
- Unidentified speaker (UNKNOWN): Powell
- 21:38
- Unidentified speaker (speaker_1): How is that possible?
- 21:40
- Unidentified speaker (speaker_0): So 3D in the chip world, it is a dimension that people have been scaling for a long time. But it's got all of these complexities. So actually, if you look at an HBM, a high bandwidth memory, it is a stack of DRAM chips. But it's a stack that has certain properties that don't make it amenable to just being taken and plonked on top of your logic die. And
- 22:00
- Unidentified speaker (speaker_0): so we don't do that today. But there are these processed technologies, and this is something that sort of you know Fractile's team has a long hinterland in around 3D stacking.
- 22:11
- Unidentified speaker (speaker_1): So you're building condos on the beach instead of beach houses to allow for more capacity. Aaron
- 22:17
- Unidentified speaker (UNKNOWN): Powell I
- 22:18
- Unidentified speaker (speaker_0): guess we're building one of those kind of uh you know new built archipelagos where everybody has a bit of beach front, french, right? It's uh strips of land, everybody's right next to a piece of coast. When you put this on top, suddenly you have this whole two-dimensional area where you can have essentially those copper wires are now not going laterally, you have millions of
- 22:37
- Unidentified speaker (speaker_0): them coming straight down.
- 22:39
- Unidentified speaker (speaker_1): When you look at differentiation between other providers, is that your differentiating factor? And how do you analyze the the specialized providers that also compete with you in this realm, like an etch? atch? Like what is the difference? differentiation?
- 22:51
- Unidentified speaker (speaker_0): Yeah, I think there's a few ways to look at this. You know, one is one is sort of philosophical, and then the other is technology today. So I think philosophically, I do think that Fractile is you know, we have a mission, right? And the mission is to advance Frontier AI capability through building better hardware. Today that's taken us over the past, you know, past couple of
- 23:11
- Unidentified speaker (speaker_0): years to this particular solution. And I think this extraordinary work that the team has done in unlocking a future where you can have the bandwidth of SRAM, but with the capacity and the scalability of DRAM, it is a world-changing kind of innovation. And it's going to have longevity as well. You know, we're we're very enthused about taking this
- 23:31
- Unidentified speaker (speaker_0): forwards over multiple generations and kind of compounding a technology advantage there. But at the same time, you know, I think with a with a mission. if the workload changes, you also want to be able to change your response. And so, you know, I think one thing which we're cautious not to become is the kind of the company that cracked 3D DRAM. We
- 23:51
- Unidentified speaker (speaker_0): want to be challenged to continue to innovate on whatever comes next. If you look at mechanistically what it does mean for us, though, over the next few years, there is this sort of challenge today. If you look at the chips. chip space where on the one hand, you have all of these HBM-based chips, and at some level it's very hard for them to diversify. Whether it's you know the
- 24:11
- Unidentified speaker (speaker_0): GPUs, TPUs, the Jalapeno chip from OpenAI, etched, MatX, they have often really brilliant kind of architectures on the chip, which allow them to do things slightly more efficiently than one another, and so on. But they are fundamentally kind of boxed in by the like top trump specs, right? You can get only so many flops and you can get only so much bandwidth.
- 24:31
- Unidentified speaker (speaker_0): The uh HBM that sits off to the side of them is exactly the same piece of silicon. It's the same module from the same vendors for each and every one of those chips. And then on the other side you've got these SRAM-based chips that have kind of high memory bandwidth, but also have this huge capacity constraint. And so if you look at, you know, if you look at
- 24:51
- Unidentified speaker (speaker_0): Grok or you look at Cerebrus, what they can do today, which is extraordinary, is allow you to run powerful models much, much faster. One of the things that I think suddenly I've always been excited about with Fractile, it was sort of what I set the company up to do four years ago, is build chips that are better at LLM inference, but in
- 25:11
- Unidentified speaker (speaker_0): particular build chips that allow us to run that inference much, much faster.
- 25:15
- Unidentified speaker (speaker_0): Because we always believed that we would get to this space where we had what we actually do have today, what's called test time compute scaling, right? Reasoning models, long-running agents that run for longer and longer to get better and better and better at what they're doing,
- 25:31
- Unidentified speaker (speaker_0): a better answer for you. And this is the kind of profile of all of the frontier workload that's sort of changed. changing the world today. You know, if you look at what happens when you turn a frontier LLM to the problem of solving a Millennium Prize problem, it will go off and generate many millions of tokens sequentially. And you
- 25:51
- Unidentified speaker (speaker_0): scale up this swarm of them as well. So there's a sort of parallel element of this. But the many, many millions of sequential token generation means you're waiting days and days and days to get one of these responses. And it's a space which somewhat tantalizingly hasn't been cracked at all by any of the silicon, any of the chips that we have today. Because there's this
- 26:11
- Unidentified speaker (speaker_0): sort of tantalizing mismatch, I guess, between the fast inference chips, the SRAM-based chips that have the ability to access memory quickly, generate tokens quickly, and the needs of this new workload, the kind of long-running agent, which needs a huge amount of capacity. because you have these really long contexts that are emerging. And so the world
- 26:31
- Unidentified speaker (speaker_0): urgently needs this kind of up and to the right thing where you have simultaneously really high bandwidth and high speed, but also high capacity. So the ability to scale to longer contexts and larger models.
- 26:43
- Unidentified speaker (speaker_1): Is this one of those where it always takes longer than you think? Or is this one where you see dramatic inflection points in the acceleration of technology?
- 26:51
- Unidentified speaker (speaker_0): I think what's exciting about this one is and this is true I think often of of sort of, you know, technological breakthroughs, you are always standing on the shoulders of giants. In our case, you know, some of the technology that we're leveraging to do, this three D integration, for instance, has actually been around for quite a long time. It's just not been turned
- 27:11
- Unidentified speaker (speaker_0): to this exact problem, to the the memory stacking challenge. And so it's a space where if you're willing to do the kind of both the architectural effort and redesign your chips to make use of this, and then you're also willing to do the kind of manufacturing operations, legwork, and you're able to persuade your partners that you're sort of not crazy along the
- 27:31
- Unidentified speaker (speaker_0): way. There is actually the possibility here for these kind of overnight, overnight transformations. Aaron
- 27:36
- Unidentified speaker (UNKNOWN): Powell
- 27:37
- Unidentified speaker (speaker_1): Can I be blunt? I was talking to a founder of a brilliant chip company last night, and he said there's a huge amount of hot air and BS in the industry. And there's a lot of marketing fluff, but there's not much actually in production. When you look at your etched or your fractiles, are we actually in production? Like how do we actually think about on the ground being in
- 27:57
- Unidentified speaker (speaker_1): production? And is he right with that statement? Aaron
- 28:00
- Unidentified speaker (UNKNOWN): Powell
- 28:00
- Unidentified speaker (speaker_0): It's a great it's a great sentiment. There's two parts. of this. You know, yes, Fractile has silicon that's gone through the foundries, has come back to us. Yes, we're running further trial runs and we'll be doing sort of full product tapeouts in a few months as well, which is quite exciting. I think the scale of production is something which is very much worth
- 28:20
- Unidentified speaker (speaker_0): challenging, right? There is a real challenge, especially for kind of new chip entrants at the frontier.
- 28:27
- Unidentified speaker (speaker_0): in accessing, for instance, TSMC capacity, HBM capacity. And I think in particular, you know, for some of these efforts, there is a race for exactly the same resource. You know, a lot of these chips today are built with the same backend partner, for example. You know, whether it's the TPU or jalapeno, MTIA from Meta, MatX
- 28:47
- Unidentified speaker (speaker_0): or Etched, you know, all of those are rumored to be using broadcast. Broadcom, for instance. Same or very similar TSMC process node, same HBM technology. And so you can see why there might be like a real kind of gold rush for that particular finite resource. And I think that means that it can be challenging to get to really meaningful scales in a
- 29:07
- Unidentified speaker (speaker_0): short space of time. And one of the things that we've sort of, I guess, always been quite focused on at Fractile is because we've had this. sort of almost unfair reach into the stack, right? The point of a back end partner is they do the kind of last mile, the physical design, they interact with the foundries, with the manufacturing side of the business. Fractile's always had that
- 29:27
- Unidentified speaker (speaker_0): in-house and we have this extraordinary team that has kind of always worked that way, you know, even before Fractile for for decades in how they've delivered their chips. And it's become a really core strategic advantage for us because it means we've both always had direct communion with our manufacturing partners. We've had to do that legwork from the outset. Aaron
- 29:45
- Unidentified speaker (UNKNOWN): Powell And
- 29:45
- Unidentified speaker (speaker_1): just so I understand the alternative is you go through Broadcom, who then have the manufacturing partner relationships.
- 29:50
- Unidentified speaker (UNKNOWN): That's right.
- 29:51
- Unidentified speaker (speaker_1): So Broadcom is the intermediary layer that most people work with that you don't. Aaron
- 29:56
- Unidentified speaker (UNKNOWN): Powell
- 29:56
- Unidentified speaker (speaker_1): And
- 29:56
- Unidentified speaker (speaker_0): that we don't.
- 29:57
- Unidentified speaker (speaker_1): What does that enable you to do then?
- 29:59
- Unidentified speaker (speaker_0): So one thing is build completely different kind of building blocks for these chips.
- 30:03
- Unidentified speaker (speaker_0): I think if you look at a Broadcom, you know, I was at a conference this summer where uh Charlie Kawas, who's you know one of the leaders there, held up a kind of Lego model of the next gen platform. And he said this is gonna be in production with six major customers in twenty twenty eight. Which is a sort of strange thing to say, because what he means is
- 30:23
- Unidentified speaker (speaker_0): something that looks identical to this is gonna be going made to with six of the major players and given different names. It'll be the TPU, MTIA, Jalapeno, you know, three,
- 30:33
- Unidentified speaker (speaker_1): Dorito 2.
- 30:34
- Unidentified speaker (UNKNOWN): Exactly.
- 30:35
- Unidentified speaker (speaker_0): And so it sort of reveals something about the structure of this market, right? Where each of those chips will have a different name. They will also be different. There are incredible chip teams inside all of those companies that come up with insanely clever kind of architectural optimizations. But in the end it's going to gonna be exactly that. same platform. It's gonna be the same stack of HBMs, the same number of chips.
- 30:55
- Unidentified speaker (speaker_0): A lot of the sort of fundamental building blocks are dictated by the back-end partner because it's a platform technology play. You know, it's a kind of you get to change the color of the paint, basically. And so for us, what it's allowed us to do is like maneuver and build these new building blocks. So the memory stacking work, for instance, is
- 31:15
- Unidentified speaker (speaker_0): incredibly complex and painful and requires convening multiple, you know, much larger supply chain partners to get them to do something that is kind of highly custom. But once you crack through that, as we have, you know, you then have an asset that nobody else has. And this was a slightly long-winded way of getting all the way to your
- 31:34
- Unidentified speaker (speaker_0): question about like volume production that also more likely. than not, has meant that you've gone and found a way to use resources that others haven't. So in our case, a different process node, maybe a slightly different DRAM node or DRAM generation as well than might be going into Frontier HBM. And so you sort
- 31:54
- Unidentified speaker (speaker_0): of find the opportunity to build gigawatts of capacity sort of down the back of the sofa with things that others other people and this sort of set of frontier platforms have already moved away from.
- 32:07
- Unidentified speaker (speaker_1): Can I ask you when we think about the kind of production line, so to speak, there, a lot of people told me that I need to ask about tapeout. What what is tape out? What do we need to know? And how how do you think about it?
- 32:17
- Unidentified speaker (speaker_0): So tapeout is the moment where the final file from your chip design process, which is actually basically uh a it's like an enormous image. We call it a GDS. GDS2 file. And the image dictates, you know, where are all of the copper traces, where are the transistors, where's the logic? That gets sent to to TSMC or to your to your logic foundry
- 32:37
- Unidentified speaker (speaker_0): partner, and they commence the process of building the mask sets, the kind of stencils that will ultimately produce the chip. And then you start doing your your first kind of trial runs, your first production runs as well, potentially. So it is a moment in time. that suddenly if you're a chip designer, is a bit of a kind of seminal handoff, right? Because at the end of this long
- 32:57
- Unidentified speaker (speaker_0): process of architecture, front-end design, verification, physical design layout, you send the file. And if there's a bug in it at this point, you know, it's too late. The mask is getting printed. And so that's why it's kind of a a you know, it's a major date on the calendar sort of thing. It's also, though, just sort of today's step step one in quite a long process,
- 33:17
- Unidentified speaker (speaker_0): right? Because that silicon then goes through the fab and you wait months for that to come back. And you have to be poised and ready then to receive that and run your bring up process.
- 33:33
- Unidentified speaker (speaker_0): To get a workload running on a single chip, on a tray, eventually on a rack as well. And so that's sort of part two, I guess, is that at Fractile, we track progress. There's tape out dates in the middle of it, but the thing that we're continually shooting for, which we call task zero, is
- 33:53
- Unidentified speaker (speaker_0): the day that a particular reference workload runs on an entire rack of fractile chips. Aaron
- 33:59
- Unidentified speaker (UNKNOWN): Powell
- 33:59
- Unidentified speaker (speaker_1): What does no one see about this process that is most proper. prominent to you? If we think about tape-out being like a seminal moment, handoff. Where is the God? No one quite understands the difficulty, the momentousness of the post-tape-out date. Aaron
- 34:14
- Unidentified speaker (UNKNOWN): Powell
- 34:14
- Unidentified speaker (speaker_1): I
- 34:14
- Unidentified speaker (UNKNOWN): think
- 34:14
- Unidentified speaker (speaker_0): well, I think that's it actually, is it's what comes next? Because there is an awful lot that you can fail to prepare yourself for if you're ultra-fixated ultra fixated on that one thing, the kind of handoff of the first. of chip to the to the fab. And so I think, you know, for us, it's continually tracking, you know, what is what is it that we're
- 34:34
- Unidentified speaker (speaker_0): actually trying to build here? We're trying to build systems that yield well. So the underlying economics are great because you know we've we've designed silicon that has the right levels of repair and redundancy. It's you know highly manufacturable. It makes a lot of sense as a product and then fits into a system that, again, yields well, is reliable.
- 34:54
- Unidentified speaker (speaker_0): And so the sort of building of a product that is manufacturable in volume, where you can then find a path to ramping as quickly as possible, that's the goal in some ways. And I think tape out is a um, I almost feel like maybe over the last few years, there's been this sort of accidental shared.
- 35:15
- Unidentified speaker (speaker_0): obsession between sort of chip startup founders and vent the venture capitalists that are backing them on this phrase because it's such a tangible moment, it's such a tangible date. But a lot of the work that we do at Fractile, you know, all of the work we do on building software, all of the work that we do on systems, and a lot of the silicon team as well is
- 35:34
- Unidentified speaker (speaker_0): focused on those parts of the chip that in a sense are kind of like intangible, ineffable, but are about what it looks like to be able to get this stuff up and running and reliable and to volume as quickly as possible.
- 35:49
- Unidentified speaker (speaker_1): When we were downstairs, you said that actually silicon may be no longer the bottleneck in the same way that it was before, and maybe data centers were or data center capacity was. Can you talk to me about that relationship and the the transition of shortage versus not? Yeah,
- 36:03
- Unidentified speaker (speaker_0): it's been, I mean, this is why the last four years have been so exhilarating because it feels like every quarter the thing that is most constraining, let's say, our ability to run more front-air models at scale in the data center has shifted. There's these underlying properties that we urgently need, like the ability to run inference much, much faster, the ability to run at longer
- 36:23
- Unidentified speaker (speaker_0): context with better economics. But then we've had this sort of, you
- 36:26
- Unidentified speaker (UNKNOWN): know,
- 36:26
- Unidentified speaker (speaker_0): if you think back three years ago, there was no question of other chips, it was an NVIDIA GPU shortage. And then I think we found ourselves in an underlying silicon and supply chain shortage, you know, acute memory shortages today for HBM and so on, which stretch years into the future, logic silicon shortages with TSMC as well. And I think
- 36:46
- Unidentified speaker (speaker_0): if you look at the space today, land and power is definitely another limitation. You know, we've had all of the kind of low-hanging fruit for industrial sites that can be retrofitted to be data centers, places where in the US you can get permitting to bring gas turbines close to spin these things up in 90
- 37:06
- Unidentified speaker (speaker_0): days. These kind of often one-time tricks have kind of all been exhausted. And so there's sort of, we're now exploring the kind of longer tail of you know harder, harder ways to find land and power to install these chips. And so on the one hand, actually it is data center capacity that is our kind of
- 37:26
- Unidentified speaker (speaker_0): fundamental bottleneck on how much compute we can build out. And so this is the kind of uh you know if you want to be completely agnostic and invest in, you know, your long AI, you know, by buy energy basically, because that is obviously the kind of most fundamental. But at the same time, it's precisely that kind of constraining
- 37:45
- Unidentified speaker (speaker_0): limit that is why it is so vital that we have the best possible chips for any given job. If you look at Frontier Labs and hyperscalers today that have a fast mode on their models, if you're running that on an HBM-based chip, you're doing so at this kind of extraordinarily inefficient point on a
- 38:05
- Unidentified speaker (speaker_0): kind of trade-off curve. On these chips, you can either be sort of slow and efficient in generating tokens or fast but deeply inefficient, sort of far fewer tokens per second per megawatt of power, for instance. So what happens if I want to take a multi-trillion Trillion parameter model and run it at thousands of tokens per second, but I have a
- 38:25
- Unidentified speaker (speaker_0): 500 megawatt data center. and I don't have enough aggregate throughput, you know, I can't serve my user base on those chips. There's a real pressure to now install a set of chips that do retain, let's say, really good aggregate throughputs, even as you squeeze them to much, much higher speeds. You know, fractile would be an example of that. And so
- 38:45
- Unidentified speaker (speaker_0): although it's land and power that may be the kind of most acute bottleneck, even ahead of kind of supply chain constraints, it's actually exactly that that kind of incentivizes chips that are much, much more fit for purpose. Aaron
- 38:58
- Unidentified speaker (UNKNOWN): Powell
- 38:58
- Unidentified speaker (speaker_1): How much of an energy deficit do we have? I know it's an incredibly macro and bold question,
- 39:03
- Unidentified speaker (UNKNOWN): but
- 39:03
- Unidentified speaker (speaker_1): when you look at energy requirements versus energy supply, what do you think the deficit looks like?
- 39:08
- Unidentified speaker (UNKNOWN): I
- 39:08
- Unidentified speaker (speaker_0): think if you look at backlogs in the US and the UK, it's this extraordinary Um, you know, the cues for grid connection will be like an order of magnitude more than there is energy in the grid. So viewed in that way, you know, and the extent to which all of those are actually legitimate projects that would get off the ground is
- 39:28
- Unidentified speaker (speaker_0): obviously something that we can we can question. But it does look like there is this staggering level of demand. And I think energy is one of these real fundamentals.
- 39:38
- Unidentified speaker (speaker_1): Going back to what we said earlier about DeepSeek and their incredible innovation on their inference stack, when you look at China's policy and regulatory environment towards energy, I mean it's green light anything. It's green light anything, including moving millions of people out of their homes because they need the sites for energy provisioning. And in the US, you have
- 39:57
- Unidentified speaker (speaker_1): NIMBYism. And in the UK, you have, you know, bats and butterflies preventing sites of scientific or natural beauty. Are we not just inherently disadvantaged in a world of energy requirements versus China?
- 40:09
- Unidentified speaker (speaker_0): I think this is one of the places where China has has somewhat stolen the march on the West.
- 40:15
- Unidentified speaker (speaker_0): And in particular, you know, these mega projects, you know, huge hydro projects, for example, they're ideal for AI compute build build-outs outs because they have this particular kind of signature where you it know might have once been kind of inconvenient to have fifteen gigawatts of power all arriving at a single site that then needs to be distributed. You know, an AI data
- 40:35
- Unidentified speaker (speaker_0): center doesn't really care where it's located. And so why not go put fifteen gigawatts of AI compute right next to this hydro plant in the middle of nowhere? And I think that there is in the US and in particular in the UK and in Europe, a lot that we can do to look a little more like that. And I think that the good news here, and this is something which,
- 40:55
- Unidentified speaker (speaker_0): you know, I I think the government in particular, you know, here and in and European government should be doing more to play up, is it's actually exactly that fact. It's that a lot of the power sources that could be brought to bear on this are, you know, i renewables can be a huge part of this story, right? You can locate a data center. you know,
- 41:15
- Unidentified speaker (speaker_0): in in Scotland next to an enormous wind farm where previously maybe there just would not have been a story as to why you wanted to have energy generation there. Aaron
- 41:24
- Unidentified speaker (UNKNOWN): Powell
- 41:24
- Unidentified speaker (speaker_1): Do you know what's really interesting for me is we we often we do a lot of Instagram now in short form. And the minute we talk about data centers and them not happening, we did it the other day with Microsoft pulling out of the NScale project. Everyone goes
- 41:37
- Unidentified speaker (UNKNOWN): good
- 41:37
- Unidentified speaker (speaker_1): Good riddance. Good riddance, you tech. bros, good
- 41:40
- Unidentified speaker (UNKNOWN): riddance.
- 41:40
- Unidentified speaker (speaker_1): And it was astonishing to see the universal backlash from bluntly moronic civilians. Can I ask you, you you saw inference as a market before so many other people? Wish I'd listened years ago. The thing that I know now that I wish I'd known twelve months ago is just the infinite demand for inference. What do you know now about the market for
- 42:00
- Unidentified speaker (speaker_1): inference that you think isn't that obvious? or you hear other people say?
- 42:04
- Unidentified speaker (speaker_0): I think the thing that remains in some ways it's a s it's a even more pronounced version of my original bet. So you know, I think four years ago the insight was sort of twofold. One is as you start to get these kind of foundation models that have been trained on everything and then generalized to anything you might turn them to. Of course, you have this
- 42:24
- Unidentified speaker (speaker_0): kind of hard logic where training becomes a fixed cost. And the deployment, you know, inference is just the use of these models to do useful things. It's the only part that matters. If you were spending more on training in the long run, like why are we even building these things? You know, you really could kind of pack up and go home.
- 42:44
- Unidentified speaker (UNKNOWN): I
- 42:44
- Unidentified speaker (speaker_1): always describe it to my mother like kind of the Olympics in your same bolt. You know, he trains in the gym, great, and then decides ah, I'm not going to actually do the race. Right. What's the point, dude?
- 42:52
- Unidentified speaker (UNKNOWN): Exactly. This is why we're here. Trevor Burrus, Jr.: It's
- 42:54
- Unidentified speaker (speaker_0): like, imagine you spent far more money building a factory than you ever actually recoup from generating something out of it.
- 43:00
- Unidentified speaker (speaker_0): The hard logic of building something with a high fixed cost is that you must believe you're going to be doing enough deployment to justify that. And so You know, I think that was sort of was was part one. And then this bet that we would have you know, what we call kind of test-time compute scaling, the advent of reasoning models, they didn't arrive until two years later, for which I'm, you know, very
- 43:20
- Unidentified speaker (speaker_0): grateful to Noam Brown and the team at OpenAI that was the first to kind of, you know, publish one of these. I think what's happening today is we continue to see this sort of vector of these uh test time compute kind of scaling laws, right? It's this stack of different ways that we pursue. greater capability. So
- 43:40
- Unidentified speaker (speaker_0): one is 10xing and 10xing the amount of compute we use to train these models. But the other today is how do we how do we pour more compute into deploying these models, into the inference side? And it's a huge part of how you get to greater capability. If you look at again the examples of solving a millennium prize problem, that Navier
- 44:00
- Unidentified speaker (speaker_0): Stokes solution was, I think, 80 hours. hours of execution from 10,000 agents in a swarm. And so you have there these kind of two axes that trace out this enormous kind of volume of token generation, right? Where I've both got 10,000 parallel agents all generating their own chains of thought.
- 44:20
- Unidentified speaker (speaker_0): And then they're running for 80 hours. And so it is so different to the kind of intuition. that we might have about an LLM, where we sort of think we ask a question and the single threaded thing will go and generate almost immediately an answer. When we bring these models to bear on the hardest
- 44:40
- Unidentified speaker (speaker_0): problems, and I think generally the hardest problems, and this is a bit of good news, you know, are also the ones that have by far the best, you know, the most positive societal impact. It don't generally it is solving solving a science problem or you know, something in medicine that we maybe haven't been able to solve before. To go and do this, where you have this
- 45:00
- Unidentified speaker (speaker_0): incredibly economically and societally beneficial thing to go solve, which is beyond the reach of kind of just applying the model and asking it one shot, you know, what do you think? What's the answer to this? Inference compute is the axis. access that we have. And the Navier Stokes example is kind of evidence of that. And so I think the thing
- 45:20
- Unidentified speaker (speaker_0): that is
- 45:20
- Unidentified speaker (UNKNOWN): What do you mean
- 45:21
- Unidentified speaker (speaker_1): inference compute is the access that we have? It's the access that we have to
- 45:24
- Unidentified speaker (UNKNOWN): do it.
- 45:25
- Unidentified speaker (UNKNOWN): It's the
- 45:26
- Unidentified speaker (speaker_0): knob that we have to turn to get greater capability. So if you imagine that at any moment in time we have in the world the best frontier model, it will come from one or other lab at uh any at point in time. And the best frontier model is extraordinarily capable. You talk to it and it's smarter than any you've spoken to before.
- 45:46
- Unidentified speaker (speaker_0): But it turns out whether when you ask it, hi, I'm working on this, you know, I'm working on this challenge, we're really trying to find a a cure for this rare disease. We're not at a place, and of course we're not at a place where it's just gonna going to say, oh, here's here it is. Like I've thought of this. It's in you
- 46:01
- Unidentified speaker (UNKNOWN): know,
- 46:01
- Unidentified speaker (speaker_0): in my in my neurons, this is fired out immediately. But the thing that we also discover is if you do throw enough compute at it, if you say, hey, how about rather than just trying to give me an answer, you propose many hundreds of candidate potential solutions that you could go and explore. And you create
- 46:21
- Unidentified speaker (speaker_0): multiple copies of yourself and you go and traverse the internet and you read the literature and you debate. with copies of yourself and you exchange information and come back to me in basically a week's time, once you've generated maybe billions of tokens in pursuit of finding an answer. You know, then a little bit like if we give a bunch
- 46:41
- Unidentified speaker (speaker_0): of smart people a lot of time to solve a problem, you do have a chance that you've come back with a solution. And we really see this, this is the key. It's what every single application
- 46:50
- Unidentified speaker (UNKNOWN): is.
- 46:51
- Unidentified speaker (speaker_0): of a frontier model to solve a hard problem is going to look like.
- 46:55
- Unidentified speaker (speaker_1): My question to you is I I am a venture capitalist, I am a investor, and I see the crunch, the debt cycles, the credit, the problems that we are hitting now and about to hit. There is not enough money in the world for the requirements of this cycle. And everything that you just said there.
- 47:15
- Unidentified speaker (speaker_1): which I agree with. And then I look and go, and then we haven't addressed the physical world and physical data
- 47:20
- Unidentified speaker (UNKNOWN): and
- 47:21
- Unidentified speaker (speaker_1): robotics and everything in between that's required there. And so I'm holding these two seemingly opposing thoughts in my mind, going, what the fuck happens now?
- 47:30
- Unidentified speaker (speaker_0): Right. Well, this is why we need continuous innovation in every part of the stack that leads to serving intelligence, right? And the good news is, you know, for all that obviously we're focused over here on chips, you know, if you just look at the algorithmic side, like model efficiency, the cost assuming no hardware change to serve a given level of intelligence
- 47:49
- Unidentified speaker (speaker_0): is coming down
- 47:51
- Unidentified speaker (UNKNOWN): precipitously quickly.
- 47:55
- Unidentified speaker (speaker_0): Yeah. And as we see longer context becoming a challenge, we see incredible innovations on how you can compress context and do sparse attention and other ways of handling this. So there is sustained innovation on what you might call kind of uh the compute efficiency of the models. You know, with the same chip at ISO compute, you know, how
- 48:15
- Unidentified speaker (speaker_0): much more intelligence, how many more solutions can I can I arrive at? And then of course we do have the same on hardware. And we're working very hard to make sure that we have that hardware effect as well. And so I think what's really thrilling, and it's the reason that I think on the whole, you know, this build-out is entirely justified, and we won't see
- 48:35
- Unidentified speaker (speaker_0): a collapse in this market, is that far from the fiber build-out of 2000-2001, we have complete full utilization of the GPUs that are powered on in data centers. We don't have dark GPU fleets because as you've seen, you know, we have this enormous number of problems as a
- 48:55
- Unidentified speaker (speaker_0): species that we still have to solve. We now have tools that we can throw at those problems. And we have many orders of magnitude, more thinking that we could be doing at sort of inference time to solve the hardest challenges if we had the compute. And so, you know, I agree with you on the one hand, it feels like we need thousands of time. times
- 49:15
- Unidentified speaker (speaker_0): more compute than we have today. We're not going to get thousands of times more data centers, and I think that's probably a good thing for the world. What we will be able to do is take, you know, roughly that kind of land and power instead of data center sites that exist today, and over the next few years, through both model innovations and hardware innovations coming
- 49:35
- Unidentified speaker (speaker_0): together, squeeze more and more and more, dramatically more, into intelligence per megawatt per rack in one of those data centers.
- 49:43
- Unidentified speaker (speaker_1): So I have Mike Cannon Bruce on the show from Atlassian, and he said to me, you know, I think the cost of inference is going to come down dramatically over the next three to five years. And that's why
- 49:53
- Unidentified speaker (UNKNOWN): we're going to be able to do it.
- 49:56
- Unidentified speaker (UNKNOWN): You should do
- 49:57
- Unidentified speaker (speaker_0): me in the next one.
- 49:58
- Unidentified speaker (speaker_1): Well, I mean you in the next one is relatively close to just me.
- 50:02
- Unidentified speaker (speaker_1): You you want me doing Russian, that really
- 50:04
- Unidentified speaker (UNKNOWN): is
- 50:04
- Unidentified speaker (speaker_1): a good one. But with the Mike Cannonbrooks theme in mind, what happens to the cost of inference?
- 50:10
- Unidentified speaker (speaker_0): So I think there is, you know, if you look at the cost, what what will a token cost in 10 years' time? It's one of those sort of questions that you know begs another question, right? Like what is the model that we're running? It's probably orders of magnitude larger than the model that we have today. And so that token is also delivering in some ways, you know,
- 50:30
- Unidentified speaker (speaker_0): substantially more capability. I think the margins and inference is something that will, you know, will fluctuate and they will accrue in different places. We will be spending far more on running models in ten years' time than we are today. You know, that part I think is is clear. But who is going to win in this space? And you know, you've got today,
- 50:50
- Unidentified speaker (speaker_0): I guess, you've got chips, you've got clouds, you've got frontier labs building frontier models, you've got folks that are building and serving open models. And I think one of the reasons why I am very long on chips in terms of where that value accrues, or being one big place where that value accrues, is actually because of the nature of
- 51:10
- Unidentified speaker (speaker_0): the challenge of placing the right chip bet. You know, it's it's got this intrinsic. kind of lag, that lag to design the chip, to get the chip out, to have it built, and then the degree to which it has to have a useful lifetime in a data center means that you don't have quite that same kind of fully efficient race, you know, potentially a race to the bottom in
- 51:30
- Unidentified speaker (speaker_0): some ways in the chip space as you do elsewhere. If you have a better idea if you can build some sustained advantage in some way. I think you see a lot of the value accrue there. I think you also see a lot of it accrue still at this frontier, you know, the construction of frontier models, which have this sort of sustained advantage over some of the
- 51:50
- Unidentified speaker (speaker_0): others. It's a costly process. It requires incredibly talented teams working with huge amounts of compute, but it is something which then delivers, you know, who doesn't want the greatest intelligence on earth? Of course, people will continue. to pay for that. And so I think the overall kind of prize pool expands. There will be kind of relative
- 52:10
- Unidentified speaker (speaker_0): winners and losers within that. But it is a it's a positive sum game in inference.
- 52:14
- Unidentified speaker (speaker_1): We just had Satcher on the show the other day, and he said that we would see a future where every company has their own specialized model with their own unique and proprietary data. And actually a cool kind of value segment would be evals and benchmarks for themselves and for their company and for their own objectives. Do you agree with that world of many specialized models on a per company basis?
- 52:34
- Unidentified speaker (speaker_1): Do you think it's it's relatively techno-optimist and wrong? How do you think about that?
- 52:39
- Unidentified speaker (speaker_0): I think there's absolutely space today in just if you look at the technology, there is positive impact in taking a model and fine-tuning it for your use cases. But it's a very it's transient a thing, right? Like generally the advantage of having something that is internally fine-tuned will disappear as soon as the next release of the next
- 52:59
- Unidentified speaker (speaker_0): open model arrives. I think that can still become a bit of a kind of yes and, right? Like as long as you have a model where this, as in a you know, a model of construction these where this stacks on top of the best frontier capability, then I can absolutely believe that there is, you know, uh an advantage to having a kind of an internal, a specialized variant of something that is
- 53:19
- Unidentified speaker (speaker_0): otherwise constructed at the frontier. But I do think this is one of the hardest questions to answer, because over time it may well be that, you know, you can only do this with the open models, for example. And if that's the case, you know, what does it mean for this sort of trade-off where is it better for me to specialize an open model?
- 53:39
- Unidentified speaker (speaker_0): or just to go all in on the thing that's been released, you know, at the frontier that is still pursuing these better compute efficiencies and by extension might actually be very cost effective for my challenge? Aaron
- 53:49
- Unidentified speaker (UNKNOWN): Powell
- 53:49
- Unidentified speaker (speaker_1): So when you look at that value stack, you said obviously chips, no shit, where you think is the most valuable. Where is the least valuable segment? Aaron
- 53:57
- Unidentified speaker (UNKNOWN): Powell
- 53:58
- Unidentified speaker (speaker_0): I think there is a danger for anybody that is working, you know. purely in the software layer. I think you know I've seen a few plays recently on kind of orchestration. There's a coming wave of heterogeneity in the chips that we're gonna have in the world. You know, GPUs, other XPUs,
- 54:18
- Unidentified speaker (speaker_0): these new chip efforts. And you know, some clouds and some software providers have set themselves up to be the kind of orchestration layer over this sort of
- 54:26
- Unidentified speaker (UNKNOWN): thing.
- 54:27
- Unidentified speaker (speaker_0): coming wave. And I think there's a world in which that kind of heterogeneity is a slightly transient thing.
- 54:35
- Unidentified speaker (speaker_0): Yes, today, you know, we have, for example, a world where NVIDIA, having acquired Grok, will deploy two chips alongside one another, you know, Ruben GPUs and Grok Groc chips, and they'll do what's called pre-filled pre-fill decode disaggregation and attention MOE disaggregation.
- 54:51
- Unidentified speaker (speaker_0): where these pieces of silicon are doing different parts of the LLM inference workload. And so there's this sort of sophisticated orchestration between the.
- 54:59
- Unidentified speaker (UNKNOWN): At the same time, it's kind
- 54:59
- Unidentified speaker (speaker_1): of like model routing but chip routing.
- 55:01
- Unidentified speaker (speaker_0): Yeah, exactly. Chip routing. And you know, I think there's a kind of um technologist view that maybe, hey, what why stop there? Why not have like seven different pieces of silicon? And maybe they're from different vendors and they're each specialized in different parts of this. And I think there's a belief that this might be a huge kind of wave of value creation will be in being specialist in how you
- 55:21
- Unidentified speaker (speaker_0): orchestrate over these. I think a lot of that could collapse back down over time. You know, that that there isn't a fundamental existential reason why we desperately need to have seven different platforms to run a workload, and there isn't really a strong economic case for kind of why we would. be able to sustain that as well. I mean, if you
- 55:41
- Unidentified speaker (speaker_0): imagine even just the NVIDIA Grok data center deployment, how many NVIDIA GPU chips do I put in there versus Grok chips?
- 55:54
- Unidentified speaker (speaker_0): If you'll give me a minute on the tech here. Oh, I will.
- 55:57
- Unidentified speaker (UNKNOWN): Is
- 55:57
- Unidentified speaker (speaker_0): a uh you know, I put the weights of my model on the Grok chip because I access them really fast on that. high bandwidth, and because every user that I'm cramming through that data center, they all want the same model weights, right? So I can kind of um amortize, I can share the cost of that much more expensive memory. I can recoup it, make it up on volume. Attention, which is the part that
- 56:17
- Unidentified speaker (speaker_0): operates over the context, isn't shared across users. My context is different to your context. And so for that piece, I'm incredibly exposed to the cost of my memory. And we talked earlier about how SRAM is far too expensive, DRAM is much cheaper. So I need to fall back to the GPU and the GPU's memory to run that piece. And so that's the logic of an NVIDIA
- 56:37
- Unidentified speaker (speaker_0): data center with these two chips in today. But as I spec that data center, how many croc chips do I put? What if my context lengths get longer over time? Suddenly I probably wanted to have more NVIDIA GPUs than I did last year, and the context lengths have got longer. How much longer will they get? Maybe the model gets larger suddenly, and actually I wish I had more grok chips. And so you're very
- 56:57
- Unidentified speaker (speaker_0): likely to end up in a place where some portion of that capacity in your data center is stranded. You're just not filling it because you've got the wrong mix.
- 57:06
- Unidentified speaker (speaker_1): That's unacceptable. And
- 57:08
- Unidentified speaker (UNKNOWN): it's completely
- 57:08
- Unidentified speaker (speaker_0): unacceptable,
- 57:09
- Unidentified speaker (UNKNOWN): right? All
- 57:09
- Unidentified speaker (speaker_0): Because you have a certain number of megawatts, you're trying to juice as much revenue as you possibly can out of them. And so this idea that you're pi placing a a fixed blend, and you're not going to be able to change that data center for the next few years, right? That's fixed, it's CapEx, it's being amortized over time. Heterogeneity is is actually a really challenging thing. And I think it's a transient response
- 57:29
- Unidentified speaker (speaker_0): to the fact that you don't today have a chip that can kind of do it all. These SRAM-based chips solve this fast inference challenge a bit, right? But they don't have the memory capacity to do a test. attention. And so we still need to bring that GPU in. They also don't have the uh flops density back to the top trumps to run the compute-intensive part of LLM inference,
- 57:49
- Unidentified speaker (speaker_0): the pre-fill portion, the prompt processing, because you've got this one piece of silicon which cost you a lot, and you've had to share the area that you might put maths in with the area that you put memory in. So you definitely haven't got as much maths as your counterparty that used HBM, who could use 100% of that chip area for the tensor cores that run matrix
- 58:09
- Unidentified speaker (speaker_0): multiplies. If you look at, let's say, what Fractile is doing, you suck the SRAM upstairs basically, and you have a you have a DRAM layer stacked on top. All of that compute die on the base is now free to put tensor cores down. You can have as many flops as anybody, you can be at Top Trump's Maxa on that. But you also get the S RAM level bandwidth from the three
- 58:29
- Unidentified speaker (speaker_0): D access, and you get the capacity from the DRAM. And so suddenly you have this kind of, you know, holy trinity where you've you've got all of these things in one. And it becomes clear that you don't need heterogeneity and all of this complex orchestration and running the risk of stranded capacity. Aaron
- 58:44
- Unidentified speaker (UNKNOWN): Powell was
- 58:45
- Unidentified speaker (speaker_1): Nvidia's acquisition of Grok and insurance by
- 58:48
- Unidentified speaker (speaker_0): there's so many fun rumors about this, right? Like I think the Grok team is an incredibly smart team. And Jensen is a smart guy that wants smart people on his team. Um they also did build a platform that I think answers or answered a a big prop challenge that you know NVIDIA faced at the time, which is how do we move into the fast inference space? You know,
- 59:08
- Unidentified speaker (speaker_0): I think Jensen Johnson is quite positive some in this space. He said some nice things about Fractile last week when he was in London. you know, and is in some ways a bit of an ecosystem player. The way that NVIDIA captures the kind of bulk of this market is that they will sometimes be deployed next to other platforms, other platforms
- 59:28
- Unidentified speaker (speaker_0): that augment parts of their capability. There is another rumor that obviously Cerebrus got to deal with OpenAI. OpenAI. I think that can be an alarming moment if you're running a multi-trillion dollar GPU company. and one of your largest customers suddenly signs on for something that you perhaps hadn't previously thought was existentially necessary, which is a fast inference platform. And
- 59:48
- Unidentified speaker (speaker_0): what's your response? And I think both deals were signed on the 24th of December, which must have been a fun time for to be a lawyer as well. Aaron
- 59:55
- Unidentified speaker (UNKNOWN): Ross Powell
- 59:55
- Unidentified speaker (speaker_1): And
- 59:55
- Unidentified speaker (UNKNOWN): so
- 59:55
- Unidentified speaker (speaker_1): you think it's a knee-jet reaction to that diversification from open OpenAI? AI? Aaron
- 59:58
- Unidentified speaker (UNKNOWN): Powell
- 59:59
- Unidentified speaker (speaker_0): I think it's a acknowledgement that some parts of this space at the time required a different solution. And this SRAM play from Grok was the right solution at that moment to be able to run LLMs faster. Today, though, we've already seen ourselves, I think, move on from this. You know, you cannot solve LLM inference with these SRAM-based chips. There
- 1:00:18
- Unidentified speaker (speaker_0): isn't enough capacity. The cost to achieve speed is too high. The constraint on the context lengths you can operate on. is very, very stark. And so, you know, I'd sort of characterize that solution as you know, it gives you sort of snappier chatbots, but it doesn't necessarily give you this thing of being able to take kind of overwhelming
- 1:00:38
- Unidentified speaker (speaker_0): frontier intelligence, these agents that run for days on end, and truncating that to be orders of magnitude faster. I think that really is where the huge prize is. And today that's not something that that provides an answer to. Aaron
- 1:00:51
- Unidentified speaker (UNKNOWN): Ross Powell
- 1:00:51
- Unidentified speaker (speaker_1): So the different players in the inference space that people know are your modals, your fireworks, are your base tens, your togethers. You would bet on one. Which one would you bet on? Aaron
- 1:01:00
- Unidentified speaker (UNKNOWN): Ross Powell
- 1:01:01
- Unidentified speaker (speaker_0): I'm friends with founders of multiple of those companies. So uh
- 1:01:04
- Unidentified speaker (UNKNOWN): But it's like children, you
- 1:01:05
- Unidentified speaker (speaker_1): always have a favourite.
- 1:01:07
- Unidentified speaker (speaker_0): You know what I I like about all of these guys is they still have the capacity to dodge and weave, right? They they haven't yet locked up huge amounts of their own capacity. They're not a neo cloud. They have the potential to bet on a new chip as it comes up. They have the potential to carve out new portions of the market. I think there's a
- 1:01:27
- Unidentified speaker (speaker_0): world in which this piece doesn't fully commoditize, right? You can be specialist in serving multimodal models, text models, fast inference. So I won't place a specific bet.
- 1:01:39
- Unidentified speaker (speaker_0): I will say that I would rather be an inference provider today, you know, a specialist in running these models, getting them up, than I would a neo cloud, actually. I think there is more mileage in being able to create the sort of business flywheel of signing up these enterprise customers that consume these tokens.
- 1:01:59
- Unidentified speaker (speaker_1): Totally get you. Will they be multi-hundred billion dollar companies? And is it an Uber lift market where winner and significantly less valuable second and third place? Or is it a GCP as your AWS market? Aaron
- 1:02:12
- Unidentified speaker (UNKNOWN): Powell
- 1:02:12
- Unidentified speaker (speaker_0): Even that's only three players, I guess.
- 1:02:15
- Unidentified speaker (speaker_0): Yeah, I think you're not going to have ten. And today you probably do have a long tail that takes you towards that. Uh And if you add the neo clouds into that, you definitely have you know, tens of players, all of whom are maneuvering around billions of dollars. Are
- 1:02:29
- Unidentified speaker (speaker_1): they multi-hundred billion dollar companies?
- 1:02:31
- Unidentified speaker (speaker_0): I think we'll have a couple of multi-hundred billion dollar companies built in that space.
- 1:02:35
- Unidentified speaker (speaker_1): Can you talk to me about the new funding round? How much did we raise?
- 1:02:39
- Unidentified speaker (speaker_0): Uh it was a seven hundred and twenty million dollar Series C.
- 1:02:43
- Unidentified speaker (speaker_1): Wow. That's quite a lot of money. How was What the process? Was it fun? Because now you're like the bellows bell of the ball.
- 1:02:49
- Unidentified speaker (speaker_0): You know, if you go back two years. you know, we were still teaching people about the word inference. And it's it makes for a it makes for a kind of trickier investor pitch, right? Because
- 1:02:59
- Unidentified speaker (speaker_1): when you educate an investor, it's just it's kind of like leave then and
- 1:03:03
- Unidentified speaker (UNKNOWN): you've got
- 1:03:04
- Unidentified speaker (speaker_0): thirty minutes, right? And uh you have to spend twenty minutes. And four years ago you had to spend ten minutes explaining what an LLM was, and then you had to spend another ten 10 explaining why you felt that training these things wasn't gonna going to be the be all and end all, and you'd deploy. I'd then spend five saying, and here's why I think speed is going to matter. We hadn't yet seen reasoning models and so
- 1:03:23
- Unidentified speaker (speaker_0): on. So compared to then, it's been very fun, I think, to be able to go straight in and talk about how we have a completely unique and extraordinary team that can do things differently to everybody else, how we have a very unique position in the supply chain and communication. can maneuver differently. I think a different set of philosophies on how you build to win in this space. You
- 1:03:43
- Unidentified speaker (speaker_0): know, that's what I get to spend my time over coffee talking about. And so I think there is a real advantage to now being in the eye of the storm in a way, in a space where I think there is an appreciation of why this is such a crucial problem to go and solve.
- 1:03:59
- Unidentified speaker (speaker_1): 720 million. Talk to me about is that a enough? And suddenly you have that in your account. What changes? Like what do you do that you couldn't do before?
- 1:04:09
- Unidentified speaker (UNKNOWN): So
- 1:04:09
- Unidentified speaker (speaker_0): suddenly there's a weight of responsibility, right? Like there's a number of folks who have staked a lot of capital on us. And I think one of our responsibilities there is to ensure that we are always having putting multiple shots on goal, as it were. So one thing that capital allows us to do is kind of distribute ourselves to look at multiple things at the same time. You know,
- 1:04:29
- Unidentified speaker (speaker_0): we're now able to start looking more generations ahead. We're able to explore more parts of the stack than we maybe maybe previously would have, you know, more innovation at the systems layer, at the networking layer, and sort of building this entire roadmap of future systems is one thing that Capital allows us to do. This is an expensive space, but I think
- 1:04:49
- Unidentified speaker (speaker_0): it's a space where the underlying capital that is needed to kind of get that chip out, get those first systems back, that's been a kind of fixed cost for the last few years. And in some sense, that's not what most of this capital is used for. I think what a lot of this is now important for is that ability to maneuver faster in the
- 1:05:09
- Unidentified speaker (speaker_0): rest of the world, in the supply chain, you know, the placing of long long-term term agreements over certain components that we need to have years into the future, and the ability, I think, to build more of that muscle of predictive power as well. You know, we can now start to think about how we run more of our own experiments, work out what might
- 1:05:29
- Unidentified speaker (speaker_0): be important in the future. And as I said, Fractile will always be a chip company, but we do need to have strong insights so that we can be more helpful to our customers. about what scales well, where model architectures might head next, how you can handle very long contexts, what it looks like to run an agent swarm versus
- 1:05:49
- Unidentified speaker (speaker_0): one very long serial agent. You know, all of these almost ML research challenges that we also get to now go and tackle. And we're also expanding geographically. So obviously we've got folks in Taiwan, we've got a team in the States. I'm very excited. that we were joined just a few weeks ago by Dan Rabinovich from Meta, who worked on a lot of their AI compute systems
- 1:06:09
- Unidentified speaker (speaker_0): there, driving our kind of systems build out. So everything from the chip upwards, you know, the rack scale systems, the networking, the integration with the data center. So it it is this very long stack. And we get to go and build parts of the business, you know, where it makes sense to. And pull in the world's best talent as we as we
- 1:06:29
- Unidentified speaker (speaker_0): go after that as well. Aaron
- 1:06:30
- Unidentified speaker (UNKNOWN): Powell
- 1:06:30
- Unidentified speaker (speaker_1): Can I ask you, speaking of investors, if you can invest in any one of your suppliers, who would it be and why?
- 1:06:36
- Unidentified speaker (UNKNOWN): So
- 1:06:36
- Unidentified speaker (speaker_0): TSMC is kind of an extraordinary business in that if you look at everything that is happening today, everything traces back to them, right? And it's sort of, I mean, this is not a this is not a scoop for your podcast, right? I think it's pretty well known that TSM. TSMC underpins all of the world's chip production. It is cheap at the price. I mean, I I would I would buy that
- 1:06:56
- Unidentified speaker (speaker_0): stock every day of the week. Uh all of the world's chips that are at the frontier, or the majority of the world's chips at the frontier are built there. They are an extraordinary business, an extraordinary team. So I think we'd we'd go in there.
- 1:07:08
- Unidentified speaker (speaker_1): If you could invest in any of your customers, who would it be and why?
- 1:07:12
- Unidentified speaker (speaker_0): So I really like some of these scrolls. scrappy neo labs. Again, I'm thinking like an investor here.
- 1:07:19
- Unidentified speaker (speaker_0): They're a little cheaper. And I think we are going to see someone
- 1:07:22
- Unidentified speaker (UNKNOWN): Which
- 1:07:22
- Unidentified speaker (speaker_1): which one?
- 1:07:23
- Unidentified speaker (speaker_0): Well, friends of mine have started Inherent in London. I don't know if you've come across them. They're looking at both, I guess, the sort of the RSI challenge, but also in particular, what it takes to build systems that automate science, automate research. And I think there is a lot of mileage. here in you know it's not quite the same as pursuing the best frontier
- 1:07:43
- Unidentified speaker (speaker_0): model that goes and tackles everything. But I think there are so many interesting questions in what it looks like to orchestrate and deploy models, build systems that are better at at solving these challenges.
- 1:07:54
- Unidentified speaker (speaker_1): If you were forced to invest in one of your competitors, final one before we do a quick fire, who would it be and why? So
- 1:07:59
- Unidentified speaker (speaker_0): I'm a big believer in 3D DRAM and one other company called DRAM. D Matrix is pursuing a different architectural direction in the same space. I think both companies are going to be very successful.
- 1:08:09
- Unidentified speaker (UNKNOWN): What was their
- 1:08:09
- Unidentified speaker (speaker_1): last prize?
- 1:08:10
- Unidentified speaker (speaker_0): I don't know.
- 1:08:11
- Unidentified speaker (UNKNOWN): Uh
- 1:08:11
- Unidentified speaker (speaker_1): are you ready for a quick fire? So what have you changed your mind on in the last twelve months?
- 1:08:16
- Unidentified speaker (UNKNOWN): Maybe
- 1:08:17
- Unidentified speaker (speaker_0): I'm not quite ready for a quick fire. Do you want to read me the quick fire? Sorry, and I'll just uh
- 1:08:23
- Unidentified speaker (speaker_0): Is Is that that not not?
- 1:08:24
- Unidentified speaker (UNKNOWN): an honest quick quickfield? fire?
- 1:08:26
- Unidentified speaker (speaker_1): Hey A no. one's ever ready for the quickfire. But um so quickfire, which we're definitely keeping that last version in. What have you changed your mind on in the last twelve months?
- 1:08:36
- Unidentified speaker (speaker_0): I think the underlying pace of capability advance in models has been extraordinary. So these maths results that we saw this summer. In terms of kind of, you know, that people talk about the truncation of their timelines, right? I think what has become clear is anything that is kind of verifiable, you know, maths and coding tends to fall into this, will be
- 1:08:56
- Unidentified speaker (speaker_0): solved in the next twelve months. We will be able to tackle any of these kind of hard, verifiable challenges with frontier models.
- 1:09:03
- Unidentified speaker (speaker_1): Who do you not have on your board that you would love to have on your board?
- 1:09:06
- Unidentified speaker (speaker_0): This is an easy one. I think Demis Asabis is both an academic hero of mine and a bit of a British hero. And I think he should uh he should come aboard.
- 1:09:15
- Unidentified speaker (speaker_1): Have you asked him?
- 1:09:16
- Unidentified speaker (speaker_0): Not yet.
- 1:09:17
- Unidentified speaker (speaker_1): Well this is a good way to do it. Demos.
- 1:09:21
- Unidentified speaker (speaker_1): What's the biggest misnomer about the chip industry today? Aaron Ross Powell
- 1:09:24
- Unidentified speaker (speaker_0): misnomas about technology businesses in general are that they're very technological and it's a very, very very human space. So much of it is about relationships, so much of it in our case is about having this extraordinary team that can collaborate across the entire stack. So it's that it's powered by humans. Aaron
- 1:09:40
- Unidentified speaker (UNKNOWN): Powell
- 1:09:40
- Unidentified speaker (speaker_1): Will NVIDIA be a $10 trillion business in five years?
- 1:09:43
- Unidentified speaker (speaker_0): Yes.
- 1:09:44
- Unidentified speaker (speaker_1): You said that pretty confidently.
- 1:09:46
- Unidentified speaker (speaker_0): It's a very positive sum game. NVIDIA today trades uh this is a soundbite, so it may be unscored, but uh I believe it trades at a lower forward multiple than Cisco after the crash, you know, after the dot com bubble burst. Um
- 1:10:00
- Unidentified speaker (speaker_1): why is that
- 1:10:01
- Unidentified speaker (speaker_0): we've talked about this bubble before it even started booming, right? We're we're so traumatized by the dot-com bubble that we've priced in exuberance right from the outset. And so you know I think we're not yet appreciating how sustained this build-out will be. And you don't have to have
- 1:10:21
- Unidentified speaker (speaker_0): incremental data centers added necessarily. You know, we talked about this advance in compute efficiency. If I have finite numbers of megawatts, I need to replace my chips every few years with better chips. And so that's a huge advantage for Fractile as a market entrant, because every chip that you see in a data center today will definitely be out of those
- 1:10:41
- Unidentified speaker (speaker_0): data centers. Six or seven years tops, many of them won't last more than a couple of years. And so that's an enormous market opportunity for the market leader. They have extraordinary support. supply chain relationships. They're going to do very well. There is then this enormous, and I would say new opportunity in being the chip vendor that serves
- 1:11:00
- Unidentified speaker (speaker_0): the needs of the frontier. And so I'm very long on Fractiles chances over that horizon as well. Aaron
- 1:11:06
- Unidentified speaker (UNKNOWN): Ross Powell
- 1:11:06
- Unidentified speaker (speaker_1): You are in charge of the UK. Congratulations. What do you change to make the UK a superpower in technology?
- 1:11:13
- Unidentified speaker (speaker_0): I think, you know, sadly, so much of what unlocks growth is unlocking energy. It is doing everything that we can to bring online more renewables, more nuclear, more sources of power that allow us to have build out of industries and matter here.
- 1:11:30
- Unidentified speaker (speaker_1): Yeah, I love Alan Chang if you use I don't know if you know Yeah, Alan.
- 1:11:34
- Unidentified speaker (speaker_1): Yeah, phenomenal. Final one. What are you most excited for about the next ten 10 years?
- 1:11:39
- Unidentified speaker (speaker_0): I think it's again, we're using these models now. What are we using them for? We're using them to solve problems we couldn't solve before. And I think we're only just starting to see that pick up and take off in terms of what it means for the real world. And so it is going to be all of the ways in which life gets a bit better. We
- 1:11:59
- Unidentified speaker (speaker_0): find cures for diseases we couldn't previously tackle. We have every education that is high quality for everybody. These things are unequivocally going to come. We have the technology to make it happen now.
- 1:12:10
- Unidentified speaker (speaker_1): I totally agree with you. I think it's so awesome. You look at like class sizes traditionally. Like forty children in a class. With AI, you have like personalized education for some that like multiple choice, some that like visual learning, at their own speed. I was shit at maths, probably unlike you. I was always too scared to put my hand up and say I don't understand. because I'd look stupid. And like having personalized learning to that
- 1:12:30
- Unidentified speaker (speaker_1): point is like one tiny example. I so agree with you. Listen, Malta, I love doing the show because I really get to learn from amazing people like you. So thank you for doing it with me, and this has been so much fun.
- 1:12:40
- Unidentified speaker (UNKNOWN): Thanks
- 1:12:40
- Unidentified speaker (speaker_0): very much, Harry. It's been very fun.
- 1:12:43
- Unidentified speaker (speaker_1): But before we leave you today, Day. founders and investors sleep on eight 8 sleep. It is the intelligent sleep system trusted by Mr. Mark Zuckerberg, Dr. Andrew Huberman, Charles LeClerc, and the Aston Martin F1 team. The pod is a smart mattress cover that fits on any bed. It cools each side to 55 degrees, heats to
- 1:13:03
- Unidentified speaker (speaker_1): 110 degrees. It even tracks your heart rate, HRV, and sleep stages without a wearable. Thank God. The new pod 6 six has a hub 50% fifty percent smaller than the last one and starts at just nineteen hundred ninety nine dollars. Visit eight sleep dot com slash two zero VC. That's E I G H T S
- 1:13:23
- Unidentified speaker (speaker_1): L E P dot com slash two zero VC. While eight sleep helps you sleep better, MongoDB helps you build better. Every serious AI company right now has the same problem. Agents need accurate context fast. Most infrastructure though. it wasn't built for it. Well, MongoDB is. Eleven Labs runs
- 1:13:43
- Unidentified speaker (speaker_1): 40 million agents on MongoDB. 75% of the Fortune 100 run their most critical apps on MongoDB. Trillions of dollars moving through MongoDB every single day. Instead of stitching together 10 different tools, you get one JSON Jason native database, Vector Search, and Voyage AI embeddings in the same system. And it runs anywhere.
- 1:14:03
- Unidentified speaker (speaker_1): no lock-in. If you're building an AI, MongoDB for startups helps you move faster with Atlas and Voyage AI credits and technical support to help you ship. Don't build agents that answer once and forget. Build agents that remember and learn from your real-time data. Go to mongoDB.com slash agents. That's That's
- 1:14:22
- Unidentified speaker (speaker_1): m-o-n-g-o-d.com M-O-N-G-O-D B.com slash slash agents. agents. While MongoDB scales the product, Framer scales the story. Framer is this incredible complete website platform, not just a builder. So teams can launch and keep improving their sites in one place. Framer is the AI website builder that helps creators, teams, and businesses
- 1:14:42
- Unidentified speaker (speaker_1): ship production-ready sites faster than ever while getting every detail right. Prompt, inspect, edit, and publish in one place at a whole new pace. Agents and humans work in tandem. Agents bring speed and scale. You bring taste, judgment, and control. The work lands on the canvas and stays editable. Build custom code
- 1:15:01
- Unidentified speaker (speaker_1): components, manage CMS content, optimize SEO, and audit for issues all in one place. Enterprise grade hosting, security, and 99.99% uptime SLAs, trusted by leading brands like Perplexity and Miro. Learn how you can get more out of your site from a Framer specialist or get started building for free today
- 1:15:21
- Unidentified speaker (speaker_1): at Framer.com slash 20VC for 30% off a Framer Pro annual plan. That's Framer.com slash 20VC. Rules and restrictions may apply.
More from The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
20VC: Jev Creator on Why Data Is More Important Than Compute | Will US Open-Source Models Catch Chinese | The Cost of Intelligence and Tokens: What No One Sees But Needs to Know with Diogo Almeida
Episode discussion: Jev Creator argues AI reliability and data quality trump compute, predicting a shift from front-end apps to background automation.
Speakers’ claims: Jev Creator (Diogo Almeida) argues that the primary bottleneck for AI automation is not compute or speed, but the reliability of intelligence, defined as the consistency of decision-making rather than system uptime. He posits a 'bitterer lesson' where task definition and data quality are more critical than algorithms or compute, which he claims have plateaued in effectiveness since GPT-4.5. Creator criticizes current market hype, benchmark-maxing, and the 'AI coworker' model, predicting a 'SaaS Palooza' where incumbents win by integrating AI into workflows rather than replacing them. He also discusses TypeSafe AI's fundraising strategy, hiring philosophy, and the geopolitical landscape of open-source models.
20VC: Jensen Huang Investigation: What Happens Now | Sequoia Leave London, Close Scout Program and End Evergreen Fund | NASDAQ Loses $2.5TRN in 24 hours | Anthropic targets $1.4TRN IPO valuation
Episode discussion: Sequoia winds down Evergreen fund, NVIDIA faces inquiry, Anthropic targets $1.4T IPO
Speakers’ claims: Speakers analyze Sequoia's strategic shifts, including exiting London and ending its Evergreen fund structure due to LP liquidity conflicts and operational complexity. They discuss the NVIDIA board's investigation into CEO Jensen Huang regarding compliance and personal conduct, while noting the Nasdaq's recent $2.5T daily loss as reactive rather than fundamental. The episode highlights Anthropic's $1.4T IPO target and OpenAI's revenue growth, contrasting public market advantages with private capital constraints. Additional segments cover AI security breaches, M&A trends, and the potential obsolescence of seat-based SaaS models like Figma in favor of agentic design.