Peter Walker details OpenRouter's data on AI inference, agentic usage shifts, and Anthropic's S1 filing metrics
Summary
Peter Walker, Head of Insights at OpenRouter, outlines the company's tracking of ~600 trillion monthly tokens, noting that agent-driven usage surpassed human usage in February 2026. He analyzes Anthropic's S1 filing, highlighting $65.7B in non-cancelable compute deals and 14x revenue growth. Walker argues that compute scarcity drives open-weight adoption and that inference provider choice significantly impacts latency and cost. He distinguishes between 'neo labs' building models and 'neo clouds' optimizing GPU infrastructure, dismissing claims of intentional model degradation by labs.
Episode sections
- 0:00 – 4:33
Introduction and OpenRouter's Data Scope
Peter Walker introduces OpenRouter's scope, processing ~600 trillion tokens monthly, and outlines three research buckets: ecosystem trends, inference layer performance, and model launch success.
- 4:33 – 9:57
OpenRouter Business Model and Stripe Acquisition
Walker explains OpenRouter's marketplace model with a single API key for 500+ models and the strategic fit with Stripe for financial infrastructure and AI spend observability.
- 9:57 – 12:50
Market Fragmentation and Agentic Usage Shift
The AI market has fragmented due to an 'agentic revolution' starting in February 2026, where agents overtook human usage in token volume, leading enterprises to mix models for cost and performance.
- 12:50 – 18:51
Inference Layer Complexity and Token Economics
Performance varies significantly by inference provider, with latency differences up to 20x for open-weight models. Subscription models hide token economics, while API usage reveals high costs.
- 18:51 – 22:36
Open Weight vs. Frontier Models and Compute Constraints
Compute scarcity drives a shift toward open-weight models from Chinese labs. Walker predicts a multiplicity of specialized models rather than a binary open vs. closed winner.
- 22:36 – 27:00
Anthropic S1 Analysis and NDR
Walker analyzes Anthropic's S1 filing, noting $65.7B in non-cancelable compute deals, 14x revenue growth, and a 900% Net Dollar Retention (NDR) figure.
- 27:00 – 31:13
GPU Inference Market Dynamics and Neo Clouds
Speakers discuss the high volume of GPU inference providers applying to OpenRouter, noting that successful 'neo clouds' require significant capital and software optimization for speed and throughput.
- 31:13 – 32:55
Neo Labs vs. Neo Clouds and Anthropic's Ecosystem
The distinction is made between 'neo labs' building proprietary models and 'neo clouds' serving others' models. Industry-specific unicorns like Harvey and Lagora fine-tune base models for specific use cases.
- 32:55 – 36:09
Distillation, Model Degradation, and Competitive Pressure
Distillation creates smaller models from larger ones, a practice labs forbid externally but engage in internally. Claims of intentional model degradation are dismissed as red herrings due to inference layer constraints.
- 36:09 – 41:36
Product Wrappers vs. Model Intelligence
For major labs, raw model intelligence remains the primary differentiator. For application-layer companies, owning the user experience and harness is critical since they do not own the underlying intelligence.
- 41:36 – 45:06
AI Adoption Divide and Cultural Barriers
A generational and structural divide exists in AI adoption, with young companies adopting agent-native workflows while large enterprises struggle with bureaucracy and cultural resistance.
- 45:06 – 46:59
Personal AI Agents and Economic Empowerment
Personal AI agents are democratizing access to executive-level assistance, effectively giving every individual a free, smart EA, contrasting with legacy enterprise tools.
- 46:59 – 47:49
Career Adaptation and Strategic Framing
Professionals are advised to transition from tactical execution to strategic framing, accepting AI-driven change and adapting to new tools like AI agents.
- 47:49 – 50:34
Recommendations and Closing Thoughts
Walker recommends 'Rest is History', The Economist's 'The Graphic Detail', and Kevin Roose's 'AGI Age', highlighting the trend of tech companies hiring economists to ground product discussions.
Assets discussed
| Asset | Kind | Stance | Context |
|---|---|---|---|
| OpenAI | private company | neutral | So, yes, of course, the labs, if you're at Anthropic or Open AI, they see all of their own data. |
| Anthropic | private company | neutral | So, yes, of course, the labs, if you're at Anthropic or Open AI, they see all of their own data. |
| DeepSeek | private company | neutral | So, like there's a ton of providers on Deep Seek. |
“Stance” is how the speakers talked about the asset, not a recommendation.
Claims worth verifying
OpenRouter sees about 600 trillion tokens per month.
Peter Walker ·4:03
How to verify: Check OpenRouter's official website or press releases for usage statistics; verify via Stripe acquisition filing disclosures if available.
OpenRouter has 506 or 507 live models at the moment.
Peter Walker ·7:42
How to verify: Visit the OpenRouter model directory and count the active models listed.
Stripe acquired OpenRouter for $7.5 billion.
Matt Cohen (implied context) ·8:05
How to verify: Check the official Stripe acquisition announcement or SEC filings related to the transaction.
Agentic usage overtook human usage in token volume in February 2026.
Matt Cohen (referencing Peter) ·18:51
How to verify: Review OpenRouter's public data reports or blog posts from early 2026 regarding usage trends.
Anthropic disclosed roughly $65.7 billion in multi-year compute deals in its S1 filing.
Peter Walker ·23:10
How to verify: Read the 'Commitments' or 'Long-term Liabilities' section of Anthropic's S-1 filing with the SEC.
Anthropic grew revenue 14x last year from a starting point of $780 million.
Peter Walker ·24:09
How to verify: Check Anthropic's S-1 filing for revenue figures in the most recent fiscal year and the prior year.
Anthropic's customers spent 900% more this year than last year (NDR).
Peter Walker ·24:33
How to verify: Review the 'Net Dollar Retention' or 'Revenue Growth' metrics in Anthropic's S-1 filing.
For some open-weight models like DeepSeek, the difference in speed between providers can be 20 times.
Peter Walker ·14:32
How to verify: Conduct independent benchmarking tests of DeepSeek models across different inference providers (e.g., CoreWeave vs. Base 10) measuring time-to-first-token.
Anthropic's compute commitments are largely with hyperscalers like AWS and Azure.
Peter Walker ·26:40
How to verify: Review the 'Compute Infrastructure' or 'Vendor Dependencies' sections of Anthropic's S-1 filing.
OpenRouter was acquired by Stripe 10 days after Peter Walker joined.
Peter Walker ·4:33
How to verify: Check Peter Walker's LinkedIn profile for join date and compare with the public announcement date of the Stripe acquisition.
OpenRouter receives more than 50 inference provider applications per week.
Peter Walker ·27:07
How to verify: Contact OpenRouter support or check their official blog/documentation for application statistics.
Neo cloud providers like Cerebrus, Core Weave, Base Tent, and Fireworks combine GPU access with software to improve model efficiency, speed, and throughput.
Peter Walker ·28:54
How to verify: Review the technical documentation and service descriptions of these specific companies on their websites.
OpenRouter processes 180 trillion weekly tokens.
Peter Walker ·38:19
How to verify: Check OpenRouter's official website, press releases, or engineering blog for traffic statistics.
Anthropic explicitly forbids external users from distilling their models, while open weight models have a gray area regarding distillation.
Peter Walker ·34:21
How to verify: Review Anthropic's Terms of Service and Acceptable Use Policy, and compare with licenses of open weight models like Llama.
OpenAI shifted its investment focus to coding after realizing Anthropic was ahead in that area.
Peter Walker ·36:42
How to verify: Review public statements from OpenAI leadership or Kevin Roose's book 'AGI Age' regarding OpenAI's strategic pivots.
Kevin Roose wrote a book titled 'AGI Age' covering the backgrounds of Dario Amodei and Sam Altman.
Peter Walker ·49:25
How to verify: Search for the book 'AGI Age' by Kevin Roose on major book retailers or library catalogs.
Executive assistants (EAs) are typically $50,000 to $100,000 jobs added on top of a senior role.
Peter Walker ·45:15
How to verify: Review job postings for Executive Assistant roles on platforms like LinkedIn or Indeed to verify salary ranges.
Do your own research
- Verify Anthropic's S-1 filing for exact figures on $65.7B non-cancelable compute deals and 900% NDR.
- Check OpenRouter's public documentation or blog posts for their methodology in tracking 600 trillion monthly tokens.
- Review recent earnings reports from major inference providers to assess capital expenditure requirements for 'neo clouds'.
- Analyze latency benchmarks for open-weight models (e.g., DeepSeek) across different inference providers to verify 20x variance claims.
- Examine enterprise AI adoption surveys to validate the generational divide in agent-native workflow adoption.
- Research TypeSafe.ai's JEV model specifications to understand the trend toward specialized, narrow-task AI tools.
- Review Kevin Roose's 'AGI Age' book and 'Rest is History' podcast for context on cultural barriers to AI adoption.
- Investigate the business models of industry-specific unicorns like Harvey and Lagora to assess fine-tuning value propositions
Full transcriptexpand
- 0:00
- Unidentified speaker: I think there's three buckets of questions that we ask.
- 0:04
- Unidentified speaker: So one is what is happening to the ecosystem writ large?
- 0:08
- Unidentified speaker: That's which models are catching hold for which tasks?
- 0:12
- Unidentified speaker: How is price per token changing?
- 0:14
- Unidentified speaker: Like these broad sort of, you'd almost imagine somebody at Wharton or Harvard writing papers on them.
- 0:19
- Unidentified speaker: Those are interesting and fun questions.
- 0:21
- Unidentified speaker: They're kind of like in the background a lot of the time.
- 0:24
- Unidentified speaker: The second kind of question we ask is about the inference layer in particular, which is how are we at OpenRouter bettering the inference layer experience for everyone in AI?
- 0:36
- Unidentified speaker: So that's latency, reliability, throughput, time to first token, all these things that we're tracking.
- 0:41
- Unidentified speaker: And how have new labs, new inference providers worked with labs to like create new experiences?
- 0:49
- Unidentified speaker: And we've we've actually under the hood done a ton of work on this new algorithm that's going to be routing between these providers that we're releasing pretty soon.
- 0:57
- Unidentified speaker: It's it's amazing.
- 0:58
- Unidentified speaker: It's so cool.
- 0:59
- Unidentified speaker: And there's like a ton of research to be done there.
- 1:01
- Unidentified speaker: And then the last layer of research is on the launches.
- 1:04
- Unidentified speaker: So obviously we have relationships with all these labs, and what they want to know is hey, open router, how did our launch go?
- 1:11
- Unidentified speaker: Who did we take users from, right?
- 1:13
- Unidentified speaker: Did it achieve the thing that we wanted it to achieve?
- 1:15
- Unidentified speaker: And it's again, I think one of the fascinating things about this industry is that that pace of change is so high.
- 1:22
- Unidentified speaker: If you don't hit the launch out of the park, you're kind of screwed until the next launch.
- 1:26
- Unidentified speaker: Like you really get one week-ish to make it go very well.
- 1:30
- Unidentified speaker: And if it doesn't go very well that week, if you don't catch the zeitgeist, you kind of have to wait till the next time you release a model to like make another splash.
- 1:39
- Unidentified speaker: So again, I think that the pressure inside these places is incredibly high.
- 1:43
- Unidentified speaker: I know friends and neighbors here in San Francisco who work around these model labs and they are just at all times because the pace of change and the pressure is just, it's really aggressive.
- 1:55
- Unidentified speaker: Welcome back to Tank Talks.
- 1:56
- Unidentified speaker: I'm your host Matt Cohen, founder and managing partner of Ripple Ventures.
- 1:59
- Unidentified speaker: And today we're welcoming back the returning guest Peter Walker.
- 2:02
- Unidentified speaker: You probably know him from Carta, where his charts on cap tables and fundraising became required reading for founders and VCs.
- 2:08
- Unidentified speaker: But Peter is now head of insights at OpenRider, sitting on top of one of the biggest real-world data sets in AI.
- 2:14
- Unidentified speaker: Trillions of tokens passing through every single day, talking about closed and open weight models.
- 2:19
- Unidentified speaker: In this episode, we dig into what the usage data actually says about the AI market.
- 2:23
- Unidentified speaker: We talk about how fast the model landscape is splintered beyond cloud or GPT, and why the real fight is often happening one layer down at the inference providers serving those tokens.
- 2:33
- Unidentified speaker: Peter breaks down the moment agents overtook humans in token volume, and why caching is the hidden factor in what drives AI cost.
- 2:40
- Unidentified speaker: We get into open weight models, how far they are behind the frontier, and what it means that the best ones are coming out of the Chinese labs.
- 2:47
- Unidentified speaker: Finally, we get Peter's take on the recently announced enthropic S1 filing, and we discuss how closely usage and compute spending line up with the revenue projections.
- 2:56
- Unidentified speaker: If you're building funding or just trying to make sense of the AI stack, this one's for you.
- 3:01
- Unidentified speaker: Now let's jump into the tank with Peter Walker from OpenRouter.
- 3:04
- Unidentified speaker: Peter Walker, welcome back to the tank.
- 3:06
- Unidentified speaker: Take two.
- 3:08
- Unidentified speaker: Glad to be back, man.
- 3:09
- Unidentified speaker: Thanks for having me, man.
- 3:10
- Unidentified speaker: Oh my God, I've been waiting for this for so long.
- 3:13
- Unidentified speaker: Now that you are officially, you know, head of insights at OpenRouter, last time we had you, you were over at Carta.
- 3:19
- Unidentified speaker: Tell us quickly, you know, Peter Walker, who are you and how the heck did you land out at head of insights at OpenRouter recently?
- 3:26
- Unidentified speaker: Well, I am a data nerd at heart, especially on the visualization and graphing side.
- 3:32
- Unidentified speaker: So that's probably the direct route into the work that I was doing at Carta, where the goal was to make better sense of this amazing data asset that talked about startup valuations and fundraising and everything on the venture capital fund side.
- 3:47
- Unidentified speaker: And that was an amazing five years.
- 3:49
- Unidentified speaker: But I was feeling really pulled.
- 3:51
- Unidentified speaker: There was a little bit of FOMO to get closer and closer to what was going on in AI.
- 3:55
- Unidentified speaker: And as I looked across the various companies in the space, like open router is such an incredible place to sit.
- 4:01
- Unidentified speaker: I feel very lucky to be here.
- 4:03
- Unidentified speaker: We see about 600 trillion tokens per month.
- 4:07
- Unidentified speaker: And from that, we are probably the biggest data set in the world on multi-model usage.
- 4:11
- Unidentified speaker: So, yes, of course, the labs, if you're at Anthropic or Open AI, they see all of their own data.
- 4:16
- Unidentified speaker: But we're one of the few players that can go across the model labs and see what models are being used for which tasks, what inference providers are actually gaining traction, and just all this cool and interesting information about, you know, the topic du jour on everyone's mind.
- 4:30
- Unidentified speaker: So again, I feel pretty lucky to be here.
- 4:32
- Unidentified speaker: Super fascinating.
- 4:33
- Unidentified speaker: And you joined just before the Stripe acquisition was announced.
- 4:36
- Unidentified speaker: Is that uh correct?
- 4:38
- Unidentified speaker: 10 days in, yes.
- 4:39
- Unidentified speaker: 10 it was on day 10.
- 4:41
- Unidentified speaker: Day 10.
- 4:42
- Unidentified speaker: What a whirlwind.
- 4:43
- Unidentified speaker: Very interesting.
- 4:44
- Unidentified speaker: Yeah, man, it was uh it was absolutely crazy.
- 4:46
- Unidentified speaker: Sometimes, you know, it's better to be lucky than good, I will say.
- 4:49
- Unidentified speaker: I don't know if all your options vested, but who knows?
- 4:52
- Unidentified speaker: But nonetheless, for people who may not be super deep into the infrastructure layer, how do you describe what open router is trying to do, what they do, why they are so core to the AI ecosystem and how you get to see everything?
- 5:06
- Unidentified speaker: Yeah, I think open router, it's a pretty simple business to describe at one level, which is it is a single API key that gives you access to 500 plus models.
- 5:15
- Unidentified speaker: So instead of being locked into we're a cloud shop or we're an open AI shop, you can use all of these different models for whatever use case they are best fit.
- 5:24
- Unidentified speaker: I think that's one side of it.
- 5:26
- Unidentified speaker: The other side, I think actually candidly, the more important side in many ways is that this isn't just about model routing, it's about inference provider routing.
- 5:33
- Unidentified speaker: So when you get into open weight models, you can serve a token from an open weight model from hundreds of different providers.
- 5:42
- Unidentified speaker: Core weave, base 10, the big hyperscalers.
- 5:45
- Unidentified speaker: These tokens come from many different places and they have vastly different performance metrics.
- 5:50
- Unidentified speaker: So they differ on cost, they differ on latency, they differ on throughput, they differ on reliability, all of these things.
- 5:56
- Unidentified speaker: So just because you are using a deep seq model, you might be having a vastly different experience than your friend using a deep seq model across the city, just because you're getting that token served from a different provider.
- 6:07
- Unidentified speaker: And so we abstract all that complexity and make sure that you are getting the fastest, cheapest, most reliable token that you can on whatever model you're using.
- 6:15
- Unidentified speaker: And I think that that inference layer, there's been a ton of chatter about the labs and the models they create.
- 6:21
- Unidentified speaker: I think there's been a little less, although growing chatter about the inference layer and what that actually means.
- 6:26
- Unidentified speaker: And that's where we spend a lot of our time.
- 6:28
- Unidentified speaker: And just so people understand the business of open router, because it kind of if you on the service level, you think of this as like a data aggregator, right?
- 6:35
- Unidentified speaker: Of all the different things that are happening around models.
- 6:38
- Unidentified speaker: But what is the underlying business that open router does a service of and how does it get paid?
- 6:44
- Unidentified speaker: Well, it's effectively just a marketplace, right?
- 6:47
- Unidentified speaker: So we connect developers, enterprises, anyone who wants to use AI with the models and providers that want to sell them AI.
- 6:54
- Unidentified speaker: And then we have a service fee on top of that.
- 6:56
- Unidentified speaker: But that's the core marketplace model is, you know, the classic network effect of more models bring in more users, more users bring in more inference providers.
- 7:05
- Unidentified speaker: And then slowly that builds and builds and builds.
- 7:08
- Unidentified speaker: I do think you, the thing that you touched on at the beginning, though, is it pretty important.
- 7:11
- Unidentified speaker: You know, our CEO Alex and the rest of the founding team has from the very beginning been incredibly interested in making as much of our data as is possible public.
- 7:23
- Unidentified speaker: And this like data transparency layer, we actually feel as though more data in this AI ecosystem can actually push the entire industry towards better outcomes rather than just saying, you know, open router wants to make more money on X, Y, and Z.
- 7:38
- Unidentified speaker: How many models are there out there?
- 7:40
- Unidentified speaker: People always try to estimate it.
- 7:41
- Unidentified speaker: How many do you guys track?
- 7:42
- Unidentified speaker: I mean, on Open Router, we have 506 or 507 live at the moment.
- 7:49
- Unidentified speaker: That is not the entire model ecosystem by a long shot.
- 7:52
- Unidentified speaker: There are thousands and thousands.
- 7:53
- Unidentified speaker: Thousands and thousands.
- 7:54
- Unidentified speaker: 100%.
- 7:55
- Unidentified speaker: Yeah.
- 7:55
- Unidentified speaker: And so for people who uh saw the headline but don't really understand the way the sausage is made, why was Stripe so excited to pay $7.5 billion for a company like OpenRouter?
- 8:05
- Unidentified speaker: It makes perfect sense as it's the new infrastructure highway for the AI age.
- 8:10
- Unidentified speaker: But give our sense of the rationale for it.
- 8:13
- Unidentified speaker: No comment on the price.
- 8:14
- Unidentified speaker: I don't know whether or not that was actually announced.
- 8:17
- Unidentified speaker: But I think that the tie-up between Stripe and Open Router makes sense for a lot of reasons.
- 8:24
- Unidentified speaker: The biggest one that I can see right off the bat is that we function in very similar ways, right?
- 8:31
- Unidentified speaker: Like Stripe is trying to abstract the complexity away from payments.
- 8:34
- Unidentified speaker: Open router is trying to abstract the complexity away from using AI.
- 8:38
- Unidentified speaker: But increasingly, those two things are becoming one and the same.
- 8:41
- Unidentified speaker: I mean, the CFOs that we talk to, AI is, if not their biggest, then their second biggest and certainly fastest growing line item on any balance sheet.
- 8:52
- Unidentified speaker: So if you are going to be a financial provider of any significant scale, you need to be seeing the tokens and understanding how they're used.
- 8:59
- Unidentified speaker: And because open router is this single API key that can go with your engineers, but with your marketers, with whoever you want, we can provide this observability layer into what are you spending on AI?
- 9:10
- Unidentified speaker: What's the actual ROI of that spin?
- 9:12
- Unidentified speaker: Which parts of your organization are using it well or poorly or need to switch models or whatever it is.
- 9:18
- Unidentified speaker: And so I think that that sort of combination was really, really attractive to the Stripe folks.
- 9:22
- Unidentified speaker: Yeah, a regular person on the street doesn't understand what tokens are in the first place, right?
- 9:26
- Unidentified speaker: They are the central currency for companies building with AI.
- 9:30
- Unidentified speaker: And as Patrick Collison mentioned, the CEO of Stripe, you know, Stripe is the economic infrastructure, right?
- 9:36
- Unidentified speaker: This AI token infrastructure is built around OpenRouter to help businesses maximize profitability by routing across different requests intelligently and spending tokens efficiently, which is the biggest argument of why the ROI has not been shown or whatever these research reports are saying is not true.
- 9:54
- Unidentified speaker: I don't know how they get their data, how it's not adding value.
- 9:57
- Unidentified speaker: But can you just talk about what you're seeing out there as we currently sit today in the market?
- 10:02
- Unidentified speaker: You've described this shift as, you know, there's cloud and chat GPT, but that's not it.
- 10:07
- Unidentified speaker: That's just not the way the market is really being perceived by all the people using everything from the closed models to the open weight models.
- 10:14
- Unidentified speaker: You know, where are we in the fragmentation process?
- 10:17
- Unidentified speaker: I'd say we're still early innings.
- 10:19
- Unidentified speaker: The way that we see the market today is there was the sort of kickoff moment in late 2022 with GPT, you know, chat with GPT 3 3.5.
- 10:29
- Unidentified speaker: And then there was this sort of latent period where there was a ton of stuff happening underneath the service, but basically a open AI was synonymous with AI usage in general.
- 10:38
- Unidentified speaker: And now that has radically changed.
- 10:40
- Unidentified speaker: And I think one of the biggest changes was actually this year.
- 10:43
- Unidentified speaker: I know it feels like it was a lifetime ago, but it actually just happened in February, where we had this shift from chatting with your AI in you know, chat or cowork or sort of back and forth fields to actually having agents implement work for you.
- 11:00
- Unidentified speaker: And that agentic revolution, candidly, has been this kickoff moment to using many, many, many different kinds of models.
- 11:08
- Unidentified speaker: So, yes, of course, many of the uh customers that we have on open router are still big users of OpenAI or increasingly clawed and anthropic.
- 11:16
- Unidentified speaker: But you might be sending your security reviews to Kimmy K3.
- 11:21
- Unidentified speaker: You might be saying, hey, for my customer service organization, these responses actually work best if we use a Deep Seek model because it's fast, it's cheap, and it's reliable.
- 11:30
- Unidentified speaker: There's a thousand different ways to like lock in different parts of your enterprise usage into different models.
- 11:36
- Unidentified speaker: But more and more people are trialing this diversity of models that they have access to.
- 11:41
- Unidentified speaker: And more and more people are finding that, you know, Cloud Opus 5.5 is an unbelievable model, but maybe I don't need that level of intelligence for everything in my organization.
- 11:50
- Unidentified speaker: Do I need it to review my next email?
- 11:53
- Unidentified speaker: Uh maybe not.
- 11:53
- Unidentified speaker: Maybe I should be spending that money a little bit more wisely on a cheaper open source option.
- 11:58
- Unidentified speaker: I don't think there's ever been a marketplace of similar kind of commodity, similar commodity, you know, supply side dynamics versus the complexity of the demand side in human history.
- 12:11
- Unidentified speaker: Think about like just food in general, right?
- 12:13
- Unidentified speaker: You know, you can buy gray triple A beef for the best steak you want, or you can buy kind of single A, you know, for processed burgers.
- 12:22
- Unidentified speaker: And that's a pretty simple dynamic of what you need from an input versus an output standpoint.
- 12:28
- Unidentified speaker: But in AI land, it is so complex.
- 12:31
- Unidentified speaker: I think you had this point about how the competition, the intensity of the competition that's happening is actually not the only competition we have.
- 12:39
- Unidentified speaker: We've got inference, compute, energy.
- 12:43
- Unidentified speaker: There's so many different levels to what makes a good decision when it comes to deciding on what models to use.
- 12:50
- Unidentified speaker: And then that has a huge ripple effect across the outcome, the ROI.
- 12:54
- Unidentified speaker: Can you just unpack that for some of our listeners to understand like how complicated this service is?
- 13:02
- Unidentified speaker: I mean, it's wildly complicated under the hood.
- 13:05
- Unidentified speaker: And again, I think when people hear the idea of model routing, the initial conception is actually fairly simple, where you say, I uh let's start with an easy enterprise customer.
- 13:15
- Unidentified speaker: We are a cloud shop.
- 13:17
- Unidentified speaker: All we do is use cloud models.
- 13:18
- Unidentified speaker: Even within Cloud, you can route to different models.
- 13:21
- Unidentified speaker: Some use Opus, some use Sonnet, some use Haiku, depending on the kinds of intelligence that you need for a specific task.
- 13:27
- Unidentified speaker: But then you look outside of Claude, and maybe you're doing that because your bill went up 14x this year.
- 13:33
- Unidentified speaker: Um, and you say, whoa, whoa, whoa, love the AI, love cloud, love enthropic, but we just like we need to get a handle on these costs.
- 13:40
- Unidentified speaker: And so you you may turn to open router and say, okay, what are the available open models that I want to use?
- 13:45
- Unidentified speaker: And where should I use them in my business?
- 13:47
- Unidentified speaker: That is the basic level of routing.
- 13:50
- Unidentified speaker: Underneath that level is this intense competition to be the provider of choice to serve you a token.
- 13:57
- Unidentified speaker: And so when you're using Claude, you basically only get tokens from one or two places.
- 14:01
- Unidentified speaker: You get it from Anthropic itself, not that common.
- 14:04
- Unidentified speaker: You get it from Bedrock or Vertex, which is AWS or Azure, like the big hyperscalers who can serve you tokens, and that's about it.
- 14:12
- Unidentified speaker: Whereas if you get an open model, because you can download the weights of that open model and serve it in a lot of different places, there are hundreds of providers that are willing to sell you tokens.
- 14:22
- Unidentified speaker: And some of them are super shady, candidly, and some of them are incredibly good.
- 14:28
- Unidentified speaker: And some of them have fine-tuned their work to work better with specific models.
- 14:32
- Unidentified speaker: So, like there's a ton of providers on Deep Seek.
- 14:34
- Unidentified speaker: Some of them, the difference in speed on a Deep Seek model can be legitimately 20 times.
- 14:40
- Unidentified speaker: Like one provider serves you a token 20 times faster than another provider.
- 14:45
- Unidentified speaker: So one of the things that's annoying about AI is, you know, if you're on Twitter or X at any point, you'll probably see in effectively back-to-back posts, somebody be like, I love Opus Fi5 because it does X, and somebody be like, I hate Opus Fi5 because it does Y.
- 15:00
- Unidentified speaker: I actually think that a lot of the reason that people have such different opinions about the same model has nothing to do with the model itself.
- 15:07
- Unidentified speaker: It has to do with the token that they're being served.
- 15:09
- Unidentified speaker: And if it is a slow or unreliable or bullshit kind of provider, you can get bad tokens and then you have a bad experience with the model, let alone if you put that in your customers-facing app and then your customers have bad experiences with your AI, that can be a really damaging part of your reputation.
- 15:26
- Unidentified speaker: So that is the complexity underneath on the routing layer towards the inference layer.
- 15:31
- Unidentified speaker: Is it's actually, I'd say, more complex on inference than it is on models.
- 15:35
- Unidentified speaker: We like to dumb everything down here for our listeners, Peter.
- 15:37
- Unidentified speaker: So when you say tokens, not at all.
- 15:39
- Unidentified speaker: Yeah, exactly.
- 15:40
- Unidentified speaker: Try to try to put this together for people.
- 15:42
- Unidentified speaker: When you say tokens, there's bad tokens, there's good tokens.
- 15:45
- Unidentified speaker: You know, think about like a carnival.
- 15:46
- Unidentified speaker: You go out to the booth, you buy $20 worth of those little chips or whatever to go into all the uh the different carney rides.
- 15:53
- Unidentified speaker: Yep.
- 15:54
- Unidentified speaker: Some rides are different than others, but some cost four, some cost five, some cost all the same.
- 15:58
- Unidentified speaker: When you say bad tokens, people just think I'm paying 20 bucks or 200 bucks a month, I'm getting access to these models.
- 16:05
- Unidentified speaker: But that's not really how the economics work.
- 16:08
- Unidentified speaker: The tokens that you're being given on those monthly subsidized plans is actually what you're paying for.
- 16:14
- Unidentified speaker: Yes.
- 16:14
- Unidentified speaker: And the reason why using Fable or Mythos to come up with a recipe for banana bread is not the best use of your tokens, explain that to listeners so they understand why it's important to choose, you know, degradated models or older models for things like those tasks.
- 16:31
- Unidentified speaker: Yeah, I think that the idea of how you are buying these tokens is actually a fundamental difference in the way you perceive the market.
- 16:38
- Unidentified speaker: And you just hit on the perfect example, which is subscriptions.
- 16:42
- Unidentified speaker: If your interactions with AI have simply been, I pay anthropic a hundred bucks a month for my Claude subscription, you really don't have a sense of how those tokens are being spent.
- 16:54
- Unidentified speaker: Or perhaps the only time you run into limitations is when you hit your usage limit for the week and you have to sit there and twiddle your thumbs for a while until it pops back on and you get a reasonable.
- 17:02
- Unidentified speaker: You go back to pen and paper, yeah.
- 17:03
- Unidentified speaker: Yeah, exactly, which is such a silly experience.
- 17:06
- Unidentified speaker: But the reason that that's happening is because they have a set level of compute.
- 17:10
- Unidentified speaker: And if they gave everybody a limited usage, it would rip through all the compute that they have and they would everyone's experience would be degraded.
- 17:17
- Unidentified speaker: So that's one way to serve tokens in AI, which is to buy subscriptions from model labs.
- 17:23
- Unidentified speaker: The other way is to pay their API prices.
- 17:25
- Unidentified speaker: And if you compare the API price to the subscription price, you look at the subscription and you say, I am getting a fantastic deal.
- 17:33
- Unidentified speaker: The subscriptions still today are fantastic deals.
- 17:36
- Unidentified speaker: We buy multiple subscriptions instead of doing the usage, as everyone should do.
- 17:40
- Unidentified speaker: They just don't know it.
- 17:41
- Unidentified speaker: 100%.
- 17:42
- Unidentified speaker: And eventually you get to a point as a business where you say, okay, I'm going to pay the API.
- 17:47
- Unidentified speaker: There's advantages to some of the API, not generally in cost, but in other ways.
- 17:51
- Unidentified speaker: And then once you get exposed to the API costs, some of those become absolutely eye-watering.
- 17:56
- Unidentified speaker: And here's the thing some of it is worthwhile.
- 17:59
- Unidentified speaker: I'm not saying that the core model labs or Anthropic or OpenAI are not worth it.
- 18:04
- Unidentified speaker: In many cases, they absolutely are.
- 18:06
- Unidentified speaker: They still have produced to date the best models, the most intelligent models, the most capable and flexible models on the market.
- 18:13
- Unidentified speaker: What I'm saying is that for a lot of use cases, you just don't need the most expensive token.
- 18:18
- Unidentified speaker: And sometimes that's flipping to a haiku model, but a lot of times that's flipping to Deep Seek or Kimmy or GLM or ZAI, or there's a hundred models.
- 18:25
- Unidentified speaker: And we haven't even gotten to the point of this is not just a competition layer between companies.
- 18:31
- Unidentified speaker: It is also, in many ways, a geopolitical competition.
- 18:35
- Unidentified speaker: And so a lot of the open weight models tend to be Chinese, and a lot of the closed weight models tend to be American.
- 18:41
- Unidentified speaker: And the back and forth there, though that's changing a little bit, the back and forth there is, you know, another point positive of like how important this shift is within AI in general.
- 18:51
- Unidentified speaker: You had a really interesting comment in February 2026 as the moment agentic usage overtook the human usage in token volume, right?
- 18:58
- Unidentified speaker: You know, that's obviously now switching with personal agents like Muse and Instinct and Grockbot over an abundance of kind of people asking for basic token usage questions.
- 19:10
- Unidentified speaker: Open weight models have driven token costs well off the frontier down dramatically, though, right?
- 19:14
- Unidentified speaker: Which is hugely advantageous for people who are able to access and build on top of these uh open weight models with uh their own workloads.
- 19:21
- Unidentified speaker: You see a lot of the work that's being done now in enterprises being switching to open weight models as well.
- 19:28
- Unidentified speaker: Where do you see this sort of open weight model versus the frontier model debate going, you know, when it comes down to the lack of compute we have and the spot prices continuing to go up?
- 19:40
- Unidentified speaker: It doesn't feel like it can continue to grow in that the other direction.
- 19:44
- Unidentified speaker: It has to go down, it has to go more open weight because compute is just not available.
- 19:48
- Unidentified speaker: I mean, I think that that we certainly believe that that is where the industry will end up.
- 19:54
- Unidentified speaker: But again, I do think that there's been a lot of painting of like open versus closed as though there must be one winner.
- 20:01
- Unidentified speaker: And I think the fundamentally at Open Router, we believe that there are going to be a multiplicity of ways to win.
- 20:06
- Unidentified speaker: And there are going to be a multiplicity of models that end up owning a specific part of the market and making that valuable.
- 20:12
- Unidentified speaker: You know, the goal of open router is to increase the number of models in the world and increase the usage of those models by different people.
- 20:19
- Unidentified speaker: Like the diversity of models is the thing that we care about.
- 20:22
- Unidentified speaker: And that I think has there are two outputs to that sort of thinking.
- 20:26
- Unidentified speaker: The first one is that, and you you spoke about it just a little bit ago around the idea of agentic usage.
- 20:33
- Unidentified speaker: There will be use cases that are not yet envisioned for which their models will be created.
- 20:39
- Unidentified speaker: Like a perfect example is JEV, a new model from this company called TypeSafe.ai.
- 20:44
- Unidentified speaker: JEV is effectively like a decision architecture that makes choices between yes and no or multiple choices incredibly quickly and incredibly cheaply.
- 20:53
- Unidentified speaker: Well, it turns out that classifying stuff as yes or no, or one of four choices, is something that every model does all the time and is it's like a core part of enterprise work.
- 21:03
- Unidentified speaker: JEV is going to be a revolution in that work for cheaper, faster, stronger, and it's going to be inserted into a ton of different models.
- 21:10
- Unidentified speaker: That wasn't available six weeks ago.
- 21:12
- Unidentified speaker: And it just popped out of someone's head.
- 21:15
- Unidentified speaker: Now there's going to be a whole world of decision models.
- 21:17
- Unidentified speaker: And oh, you've created a whole new category.
- 21:20
- Unidentified speaker: We are still so early.
- 21:21
- Unidentified speaker: There will be more new, innovative, cool things to try.
- 21:25
- Unidentified speaker: And it just won't, it's fundamentally, it just won't all happen within Open AI or within Anthropic.
- 21:30
- Unidentified speaker: Also on the compute side, right?
- 21:32
- Unidentified speaker: Someone's going to squeeze more out of these GPUs, these H100s, you know, these old degradated ones.
- 21:38
- Unidentified speaker: And, you know, the kernel will be continuing to be performance improvements.
- 21:42
- Unidentified speaker: I think that if there is a bubble in AI, I think it is fundamentally distinct from many other quote unquote bubble periods in history.
- 21:51
- Unidentified speaker: You know, if you look back to the dot-com boom, the bubble in that sense was the overcreation of fiber, which was not used by the companies at the time because they didn't have any actual customers or revenue.
- 22:01
- Unidentified speaker: That is the exact opposite of the situation we're in now, where it feels that everyone is compute constrained and that if there were more GPUs, we would use more GPUs.
- 22:10
- Unidentified speaker: I do fundamentally believe that's the case.
- 22:12
- Unidentified speaker: That doesn't mean that you can't get over your skis and that demand and supply might be out of whack for a period.
- 22:18
- Unidentified speaker: So, like there isn't a world, there's there are many worlds in which there is a recession and AI is sort of hit as an industry, but there is no world where we go back to a pre-AI economy.
- 22:29
- Unidentified speaker: Like that is gone, and AI will be a fundamental part of the technology landscape for from now until eternity.
- 22:36
- Unidentified speaker: It just has to be.
- 22:37
- Unidentified speaker: Yeah, absolutely.
- 22:38
- Unidentified speaker: I mean, it's so awesome to have this conversation with you at a time when the anthropic S1 has dropped because you finally get to look behind the kimono on a lot of the things that people were prophesizing around the business.
- 22:49
- Unidentified speaker: So, yes, Peter, you know, open router tracks real token volumes and spend across multiple models.
- 22:54
- Unidentified speaker: And with Anthropics S1 showing, you know, their revenue numbers, their gross margin, their compute commitments.
- 23:02
- Unidentified speaker: I mean, where do you want to start with this?
- 23:03
- Unidentified speaker: That's the most eye-popping for you.
- 23:05
- Unidentified speaker: I mean, I think the compute to me was massively understated.
- 23:10
- Unidentified speaker: I think it disclosed roughly 657 billion in multi-year compute deals.
- 23:15
- Unidentified speaker: Much of it is non-cancelable.
- 23:16
- Unidentified speaker: I can't believe that.
- 23:18
- Unidentified speaker: Um, but what was the biggest sort of few numbers in there that stood out to you from your scient open router?
- 23:24
- Unidentified speaker: Look, uh, I mean, I think it might have been the most fascinating S1 ever dropped.
- 23:29
- Unidentified speaker: I know that's probably not a difficult race because S1s tend to be dry, but this one was outrageous.
- 23:35
- Unidentified speaker: I mean, on the let's start with compute.
- 23:38
- Unidentified speaker: You're right to say that there are these hundreds and hundreds of billions of dollars of non-cancelable compute contracts.
- 23:44
- Unidentified speaker: And in some ways, the way that they are accounted for in the balance sheet can, you can have a lot of different opinions about where the actual profitability of the business lies today.
- 23:54
- Unidentified speaker: I think the trend, though, is pretty obvious, which is they are incredibly profitable on inference today.
- 24:00
- Unidentified speaker: And it's just a question of whether or not they can fulfill those commitments long term.
- 24:03
- Unidentified speaker: If the token growth and their place in the ecosystem remains, I don't think they're gonna have any trouble doing that.
- 24:09
- Unidentified speaker: So that on one side, like, yes, the compute commitments are eye-watering, but it's a business that grew 14x last year from a position of 780 million as a starting point.
- 24:21
- Unidentified speaker: Like that level of growth at that scale has never been seen before in capitalism.
- 24:26
- Unidentified speaker: It is outrageous.
- 24:28
- Unidentified speaker: For a company that's only three years old.
- 24:30
- Unidentified speaker: So, like, you know, it's uh I think it's a bet worth taking.
- 24:33
- Unidentified speaker: The other stuff that popped out to me, their customers are spending 900% more this year than they did last year if they had signed last year.
- 24:41
- Unidentified speaker: 900% in DR.
- 24:43
- Unidentified speaker: I don't believe it though.
- 24:44
- Unidentified speaker: I think that was a mistake in time.
- 24:46
- Unidentified speaker: As you can see, there's a lot of companies that didn't turn off API usage.
- 24:50
- Unidentified speaker: They didn't turn, they didn't figure out exactly how to get access to some of these open weight models, local models.
- 24:57
- Unidentified speaker: I don't know if the competition was as fierce as it is today.
- 25:00
- Unidentified speaker: You yourself are saying that more people are switching to open versus closed.
- 25:04
- Unidentified speaker: I think the the quote of technology.
- 25:06
- Unidentified speaker: That doesn't mean that that doesn't mean that they're not using Anthropic more, though.
- 25:09
- Unidentified speaker: Again, I think this is a false distinction.
- 25:12
- Unidentified speaker: You can use Anthropic more and also increase your model portfolio to include many open weight models.
- 25:19
- Unidentified speaker: Now, there are some businesses, some businesses for whom the goal is to lower the amount of money they're spending on AI per month.
- 25:27
- Unidentified speaker: Those businesses exist, and I'm not here to say whether or not that's a good choice.
- 25:31
- Unidentified speaker: Like it's an individual enterprise choice.
- 25:33
- Unidentified speaker: I'm sure that they're making smart financial decisions in general in those cases.
- 25:37
- Unidentified speaker: There are many, many more businesses who are saying we want to have exposure to different kinds of intelligence and we want to slow the growth of our AI spending, but we don't necessarily want it to turn negative.
- 25:49
- Unidentified speaker: And I think that distinction is pretty important when you get to something like a 900% NDR number, which whether or not you think that this year's NDR number will be 900%, which I would be pretty shocked by candidly, the fact that it existed for one year is outrageous.
- 26:04
- Unidentified speaker: Yeah, I agree.
- 26:05
- Unidentified speaker: Absolutely.
- 26:05
- Unidentified speaker: It doesn't make any yeah.
- 26:07
- Unidentified speaker: No, you're saying that the enterprise spend for things that are currently not adding value to enterprise bottom lines, but are being tested to see where they can add value is not going away.
- 26:18
- Unidentified speaker: And if that number ends up at 400 or 500, it's still, you know, earth shattering, to say the least, right?
- 26:24
- Unidentified speaker: Yeah, it's outrageous.
- 26:25
- Unidentified speaker: The other, the other big thing that came out of this S1, I think one of the, I mean, uh, you know, there's a thousand talking points that came out of it, but one of the biggest in my mind is just this idea that like where is your next token going to be served from?
- 26:40
- Unidentified speaker: Like if you have these hundreds and hundreds of billions of dollars of committed compute spend, the vast majority of that, for at least for the anthropic side, is with the hyperscalers.
- 26:48
- Unidentified speaker: And it's just a wither the neo clouds, like where does this layer of inference go?
- 26:54
- Unidentified speaker: Are there going to be tons and tons of successful businesses built on renting and selling access to GPUs?
- 27:00
- Unidentified speaker: Or is that where you could see the bubble really popping?
- 27:04
- Unidentified speaker: I think that's a big question at OpenRouter.
- 27:07
- Unidentified speaker: We get more than 50 inference providers applying to be on Open Router a week.
- 27:13
- Unidentified speaker: Wow.
- 27:14
- Unidentified speaker: We don't accept nearly any of them because some of them are small, some of them it takes a ton of time to like investigate where this these tokens and these GPUs are coming from.
- 27:23
- Unidentified speaker: But it feels like everyone and their mother is opening a GPU business.
- 27:27
- Unidentified speaker: Maybe we're at we have too many picks and shovels for this, for this new uh you know, gold rush.
- 27:32
- Unidentified speaker: I don't know.
- 27:32
- Unidentified speaker: May I I think that's the one place where maybe you could poke some holes.
- 27:35
- Unidentified speaker: But again, if things continue to grow this quickly, it won't be a problem.
- 27:38
- Unidentified speaker: You're saying people are just strapping a bunch of Mac minis together and saying, hey, sign up for some compute here.
- 27:43
- Unidentified speaker: We got great computers.
- 27:44
- Unidentified speaker: There are some providers who are doing the crypto mining version of it where they're like, no, we're just gonna have remote access to your MacBook and we're gonna have this distributed network of MacBooks.
- 27:54
- Unidentified speaker: It's crazy.
- 27:55
- Unidentified speaker: Like all sorts of GPU access, you know.
- 27:57
- Unidentified speaker: I think the uh people listening probably don't even understand what a neo cloud does from a business model.
- 28:02
- Unidentified speaker: Maybe explain like why they are important, why it's not something that a mom and pop can just open up in their bakery in the back room.
- 28:09
- Unidentified speaker: Like this is a heavy capital expenditure business, sophisticated software that typically runs on top of it, not many offering bare metal access.
- 28:17
- Unidentified speaker: Why is this something that people are trying to get onto open router for?
- 28:21
- Unidentified speaker: Why it's a good business for now and how hard it is to get spun up?
- 28:25
- Unidentified speaker: It's a it is deceptively quote unquote easy to spin up, but it actually isn't at all.
- 28:31
- Unidentified speaker: So you're right to say that there are very few companies whose business model is to buy GPUs and then just go and resell access to those GPUs, bare metal.
- 28:39
- Unidentified speaker: Like that's it.
- 28:40
- Unidentified speaker: That doesn't happen that often.
- 28:41
- Unidentified speaker: Far more common is a neo cloud like Cerebrus, Core Weave, Base Tent, Fireworks, all these different people who say, what we've done is we've combined access to GPUs, which is the capital commitment part, very capital intensive.
- 28:54
- Unidentified speaker: Those are expensive.
- 28:55
- Unidentified speaker: And we've developed software to run models more efficiently, more reliably, better speed, better throughput on our GPU cluster specifically.
- 29:07
- Unidentified speaker: So not only can you serve DeepSeq from this GPU cluster, it will be better in some meaningful way than serving a token of GP of DeepSeq from another cluster.
- 29:16
- Unidentified speaker: And there's a whole different, there's a ton of different aspects by which better can be achieved.
- 29:21
- Unidentified speaker: Oftentimes they start with speed, though.
- 29:23
- Unidentified speaker: That's the most important thing.
- 29:25
- Unidentified speaker: Then you have capacity constraints.
- 29:27
- Unidentified speaker: So the reason why we have so many inference providers on OpenRouter and why we like measure them basically every 30 seconds on all of these different metrics is that you might be a small neo cloud who can serve 500 billion tokens a day.
- 29:42
- Unidentified speaker: But as you get close to the edge of that, some of the performance metrics might start to degrade and we have to fail over to a different cloud that has more capacity.
- 29:50
- Unidentified speaker: And so there's this weird mix of capital intensity.
- 29:54
- Unidentified speaker: I mean, Matt, you actually might know this better than me.
- 29:57
- Unidentified speaker: I feel like one of the jobs that is most important in the new AI economy is the deal makerslash understander of interest rates on like long-term capital commitments in a way that like was never actually useful in SaaS.
- 30:13
- Unidentified speaker: Like you need deal guys who understand compute and understand how to like buy compute and sell it.
- 30:18
- Unidentified speaker: And then you need people who can actually make this inference cloud run an open weight model better, faster, cheaper than the other inference clouds.
- 30:26
- Unidentified speaker: And it is a hyper competitive space.
- 30:28
- Unidentified speaker: Not only that, you need capital markets access, you need energy access, you need securing long-term power and energy supply, you need an infrastructure team, you need physical, you know, building builders.
- 30:41
- Unidentified speaker: There are so many things.
- 30:42
- Unidentified speaker: And then you need 100% uptime, right?
- 30:44
- Unidentified speaker: To maintain this, you know, large-scale infrastructure project.
- 30:47
- Unidentified speaker: There is not any job part of building a neo cloud that is easy.
- 30:52
- Unidentified speaker: No.
- 30:53
- Unidentified speaker: And what will happen the easy part is getting people to maybe pay you for it because everyone wants it.
- 30:58
- Unidentified speaker: Maybe, but uh, even at the beginning, that like looks like easy revenue, but it really isn't.
- 31:04
- Unidentified speaker: And eventually, you know, a lot of what's happening now is that there's this weird mixed area between neo lab and neo cloud.
- 31:12
- Unidentified speaker: Explain the difference.
- 31:13
- Unidentified speaker: Yeah.
- 31:13
- Unidentified speaker: Neo labs are companies that are building their own models and trying to like get into the model making game.
- 31:20
- Unidentified speaker: Neo clouds are companies that are serving other people's models or helping you fine-tune your own.
- 31:25
- Unidentified speaker: Right.
- 31:25
- Unidentified speaker: There are many neo labs who will eventually transition into neo clouds because one of the capabilities or one of the requirements of building a neo lab is you need access to GPUs in order to run your own models.
- 31:38
- Unidentified speaker: Well, can we double-click on the Neo Labs, though, in relation to Anthropic?
- 31:41
- Unidentified speaker: Because in their S1, they also said there was a lot of Neo Labs that were using the compute of Anthropic to train their own models, to distill probably some of Anthropic's models, and then create their own smaller models.
- 31:54
- Unidentified speaker: You know, you've got very industry-specific unicorns that are now coming up with their own models.
- 32:00
- Unidentified speaker: You've got Harvey, you've got Lagora, you've got Deal, you've got all these companies coming up with their own models.
- 32:06
- Unidentified speaker: They likely used Anthropic or OpenAI to train and create those models.
- 32:10
- Unidentified speaker: And now they're fine-tuned for a very specific use case with their own harnesses built on top of them.
- 32:15
- Unidentified speaker: In Anthropics S1, they also said there was probably a good percentage of people using just maybe other overseas state actors using them to distill their own models to create these, you know, open source, uh, open weight models that uh, you know, the Chinese have obviously led the race in.
- 32:31
- Unidentified speaker: What are your thoughts on where that takes, you know, the model industry at large?
- 32:36
- Unidentified speaker: And how do you think this will impact anthropic?
- 32:38
- Unidentified speaker: Because I'm not sure in their S1 if they actually wrote how much their internal compute was actually factored into their compute efficiency improving.
- 32:46
- Unidentified speaker: Yeah.
- 32:47
- Unidentified speaker: I don't know on the latter question whether or not they lay that out in the S1.
- 32:52
- Unidentified speaker: I think in general, there's like two concepts to work through here.
- 32:55
- Unidentified speaker: One is fine tuning and one is distillation.
- 32:58
- Unidentified speaker: So on the fine tuning side, that is typically using either an open weight or with permission, some of the closed weight models, and saying, let's use Lagora as an example.
- 33:09
- Unidentified speaker: What we're gonna do is we're gonna take this base model, which does pretty well, and we're gonna feed it a ton of our data and our answers and our like ways of working, and we're gonna make it what we would call the best model for legal work on the in the world.
- 33:22
- Unidentified speaker: And that is fine-tuned based on our proprietary data with a base model that's pretty intelligent.
- 33:28
- Unidentified speaker: That's great.
- 33:29
- Unidentified speaker: And then they use that within their application to their customers.
- 33:33
- Unidentified speaker: So are they actually like branding it as a Lagora model and letting other people use it?
- 33:38
- Unidentified speaker: No.
- 33:38
- Unidentified speaker: It is the sort of secret sauce why you would sign up for Lagora versus Harvey versus other places.
- 33:44
- Unidentified speaker: Fine tuning is happening all over the place.
- 33:46
- Unidentified speaker: Fine tuning can result in massive increases in accuracy, reliability, or capability based on a lot of different vectors.
- 33:53
- Unidentified speaker: I think fine tuning is here to stay.
- 33:56
- Unidentified speaker: The second thing is distillation.
- 33:57
- Unidentified speaker: So this is the ability to take, say, a big model, anthropic Opus 5.5, and run it again and again and again and again and again, literally hundreds of thousands, if not millions of times.
- 34:10
- Unidentified speaker: Check the answers from that output, the telemetry that that model is actually giving you, and build your own version of that model based on a distilled or smaller version of Opus 5.5.
- 34:21
- Unidentified speaker: The labs basically, all of the closed labs explicitly forbid this.
- 34:25
- Unidentified speaker: The open weight models don't encourage it all the time.
- 34:30
- Unidentified speaker: There's a little bit of a gray area.
- 34:31
- Unidentified speaker: Like some are cool with distillation, many are not, etc.
- 34:34
- Unidentified speaker: What is happening inside these labs, though, is very clearly they are distilling their own models to release new variants.
- 34:41
- Unidentified speaker: So like much of the anthropic model mix right now is probably a distilled version of their best model that we don't have access to because it's what about model degradation?
- 34:51
- Unidentified speaker: Degradation of their own models.
- 34:54
- Unidentified speaker: I think this is a complete red herring.
- 34:56
- Unidentified speaker: People have pointed to the idea that the labs are quote unquote nerfing their own models.
- 35:01
- Unidentified speaker: Like Opus Fy 5 works great for a month, and then they like remove somehow its intelligence or make it work way slower or whatever.
- 35:08
- Unidentified speaker: We feel that when we're using it, I mean I don't think that's happening as a direct result of lab action.
- 35:14
- Unidentified speaker: They're not sitting there behind the seats going, what do you think it's happening?
- 35:18
- Unidentified speaker: Just act the tokens?
- 35:19
- Unidentified speaker: I think it's a lot of it is on the inference layer and you're being served slower or less reliable tokens.
- 35:24
- Unidentified speaker: And I also think that this is the most competitive industry, maybe in the history of capitalism in terms of self-cannibalization, right?
- 35:33
- Unidentified speaker: Like when you spend time, effort, and money to create Opus V, you know in six weeks you're gonna need to release Opus 5-5.
- 35:41
- Unidentified speaker: So, like, why would you degrade something that's dying?
- 35:43
- Unidentified speaker: Imagine Nike having to produce a new shoe every six weeks.
- 35:46
- Unidentified speaker: Yeah, but it also like if the old shoes keep value, old models don't.
- 35:51
- Unidentified speaker: Like nobody uses Sonic 3 anymore.
- 35:53
- Unidentified speaker: The secondary market is just not going up.
- 35:55
- Unidentified speaker: You're right.
- 35:56
- Unidentified speaker: It's not.
- 35:56
- Unidentified speaker: And you have to run as fast as possible to maintain share, let alone gain share.
- 36:00
- Unidentified speaker: It is like, it's crazy competitive, man.
- 36:03
- Unidentified speaker: So if raw intelligence becomes less differentiating from many use cases, how important is the surrounding product and harness becoming?
- 36:09
- Unidentified speaker: You know, are these labs even investing enough there, or are they still mostly focusing on these model things?
- 36:14
- Unidentified speaker: Obviously, Claude Code and cowork and things like that are great, but with personal agents taking over with Muse and Instinct, like where do you think this goes?
- 36:23
- Unidentified speaker: I think it's a massive new vector of competition.
- 36:26
- Unidentified speaker: Whether or not they're quote unquote investing enough, I think is in the eye of the beholder.
- 36:30
- Unidentified speaker: So for instance, OpenAI for many months was investing in all sorts of different things.
- 36:35
- Unidentified speaker: Sora and uh like image and video, like app location layer stuff and like whatever.
- 36:42
- Unidentified speaker: And then they looked around and went, oh, anthropic is ahead of us on coding.
- 36:46
- Unidentified speaker: And that's the thing that matters.
- 36:47
- Unidentified speaker: This is not the first time this has happened to OpenAI.
- 36:49
- Unidentified speaker: This is why Dario left to start Anthropic.
- 36:52
- Unidentified speaker: Yeah.
- 36:52
- Unidentified speaker: So, you know, read Kevin Roos's book.
- 36:54
- Unidentified speaker: I think it's uh it's quite good.
- 36:56
- Unidentified speaker: The thing there is like you can always make the case that they should be investing more in product.
- 37:00
- Unidentified speaker: Product definitely matters.
- 37:01
- Unidentified speaker: Like these coding harnesses matter.
- 37:03
- Unidentified speaker: Your familiarity with them matters.
- 37:05
- Unidentified speaker: But if a model is smart enough, if the jump in the intelligence is smart enough, many people will switch no matter what harness they're using or what is the application around it.
- 37:16
- Unidentified speaker: So my instinct is that especially for the two main labs, increases in model intelligence is more important than the app wrapper around it.
- 37:25
- Unidentified speaker: But I might be wrong about that.
- 37:27
- Unidentified speaker: For other labs, I think it's very important.
- 37:30
- Unidentified speaker: And obviously, if you're building an application layer company, it's like it is the whole thing, right?
- 37:34
- Unidentified speaker: You are not owning your own intelligence necessarily there.
- 37:37
- Unidentified speaker: You are owning the user experience.
- 37:39
- Unidentified speaker: And I think that those personal assistance, those personal agent use cases, instinct, dot, et cetera, are probably the first time where a lot of people will get introduced to AI that's incredibly powerful.
- 37:52
- Unidentified speaker: You don't even know what model is powering it.
- 37:54
- Unidentified speaker: It doesn't actually matter.
- 37:55
- Unidentified speaker: It's just this is my AI agent and it's really quite good.
- 37:58
- Unidentified speaker: Can you talk about your seat at uh OpenRouter with the head of insights role?
- 38:02
- Unidentified speaker: You know, in terms of like the questions you can ask and the insights that you can derive must be like drinking from a fire hose, like on deciding which ones to write about, which ones to go deeper on.
- 38:13
- Unidentified speaker: You know, you you get 70 trillion weekly tokens flowing through open router.
- 38:17
- Unidentified speaker: How do you know what questions to ask?
- 38:19
- Unidentified speaker: 180 weekly trillion trillion weekly tokens at this point.
- 38:23
- Unidentified speaker: It's yeah, it's the the growth is outrageous.
- 38:25
- Unidentified speaker: I think there's three buckets of questions that we ask.
- 38:29
- Unidentified speaker: So one is what is happening to the ecosystem writ large?
- 38:33
- Unidentified speaker: That's which models are catching hold, for which tasks, how is price per token changing?
- 38:38
- Unidentified speaker: Like these broad sort of you'd almost imagine somebody at Wharton or Harvard writing papers on them.
- 38:44
- Unidentified speaker: Those are interesting and fun questions.
- 38:46
- Unidentified speaker: There's kind of like in the background a lot of the time.
- 38:49
- Unidentified speaker: The second kind of question we ask is about the inference layer in particular, which is how are we at OpenRouter bettering the inference layer experience for everyone in AI?
- 39:01
- Unidentified speaker: So that's latency, reliability, throughput, time to first token, all these things that we're tracking.
- 39:06
- Unidentified speaker: And how have new labs, new inference providers worked with labs to like create new experiences?
- 39:13
- Unidentified speaker: And we've we've actually under the hood done a ton of work on this new algorithm that's going to be routing between these providers that we're releasing pretty soon.
- 39:22
- Unidentified speaker: It's uh it's amazing.
- 39:23
- Unidentified speaker: It's so cool.
- 39:24
- Unidentified speaker: And there's like a ton of research to be done there.
- 39:26
- Unidentified speaker: And then the last layer of research is on the launches.
- 39:29
- Unidentified speaker: So obviously we have relationships with all these labs, and what they want to know is hey, open router, how did our launch go?
- 39:36
- Unidentified speaker: Right.
- 39:36
- Unidentified speaker: Who did we take users from, right?
- 39:38
- Unidentified speaker: Did it achieve the thing that we wanted it to achieve?
- 39:40
- Unidentified speaker: And it's again, I think one of the fascinating things about this industry is that that pace of change is so high.
- 39:46
- Unidentified speaker: If you don't hit the launch out of the park, you're kind of screwed until the next launch.
- 39:51
- Unidentified speaker: Like you really get one week-ish to make it go very well.
- 39:55
- Unidentified speaker: And if it doesn't go very well that week, if you don't catch the zeitgeist, you kind of have to wait till the next time you release a model to like make another splash.
- 40:04
- Unidentified speaker: So it's again, I think that the pressure inside these places is incredibly high.
- 40:10
- Unidentified speaker: I know friends and neighbors here in San Francisco who work around these model labs and they are just burnt out at all times because the the pace of change and the pressure is just it's it's really aggressive.
- 40:22
- Unidentified speaker: You're like the the Nielsen TV cable data analytics, you know.
- 40:26
- Unidentified speaker: As soon as uh the next episode drops, what's how'd we do?
- 40:29
- Unidentified speaker: We got to tell the advertisers.
- 40:31
- Unidentified speaker: We like to say internally we are the nerdy, neutral, enthusiastic weather people of AI.
- 40:36
- Unidentified speaker: Yes.
- 40:37
- Unidentified speaker: You know, I just gotta ask you, Peter, you know, looking out, maybe I know it's hard to say, six months, 12 months post-anthropics IPO.
- 40:46
- Unidentified speaker: You've got all these different models dropping, all the usage is going berserk.
- 40:50
- Unidentified speaker: There's still this ROI cap uh conversation that people are struggling with in day-to-day life.
- 40:56
- Unidentified speaker: And I know the power of it.
- 40:57
- Unidentified speaker: We use it internally, we build our own systems, we run our own local models.
- 41:01
- Unidentified speaker: I know it will eventually be absorbed across the rest of the world.
- 41:05
- Unidentified speaker: But where do you think we end up getting this huge unlock?
- 41:10
- Unidentified speaker: Is it on the personal side?
- 41:12
- Unidentified speaker: Is it the cost side?
- 41:13
- Unidentified speaker: Is it just the next generation of people who are AI native, AI pilled, becoming a bit more senior?
- 41:19
- Unidentified speaker: Because it feels like there's we're in this sort of tug of war still, where the old guard does not understand or appreciate the power of this intelligence.
- 41:27
- Unidentified speaker: They want to do it, but they just don't want to spend the money on it.
- 41:31
- Unidentified speaker: And the younger generation is like powering through with all these personal agents and use cases.
- 41:36
- Unidentified speaker: You hit the biggest divide, which is generational.
- 41:39
- Unidentified speaker: And not just generational in terms of the actual individual person, but generational in terms of the company structure.
- 41:44
- Unidentified speaker: It is just far easier to work in an agent-native way at a young company who was born in this era.
- 41:52
- Unidentified speaker: It you just have to fight far less against the accumulated crud of bureaucracy.
- 41:58
- Unidentified speaker: And that isn't a negative on the bureaucratic companies.
- 42:01
- Unidentified speaker: It is effectively inevitable that when you have companies that are thousand plus people, there by definition must be bureaucracy.
- 42:09
- Unidentified speaker: There must be ways of working that have worked for a long time, and changing them is hard.
- 42:13
- Unidentified speaker: So I think that that's one of the reasons why the diffusion across these layers, you know, there seems to be a bubble in San Francisco and a smaller bubble in New York and a couple other places.
- 42:22
- Unidentified speaker: And then it's like, you know, the disfusion, the dispersion has not gotten there yet.
- 42:26
- Unidentified speaker: So what do I think is going to happen in a year?
- 42:29
- Unidentified speaker: I do think that there is a value to increasing intelligence in your organization that these models give you access to that will be very difficult to resist over the long term.
- 42:40
- Unidentified speaker: But I candidly think personally, it's still going to be a little disappointing how far AI has diffused into the broader economy, even a year from now, because it's not a technological problem.
- 42:55
- Unidentified speaker: It's a pure person-to-person cultural thing.
- 42:57
- Unidentified speaker: And that is, you know, if you're at, I don't know, Johnson and Johnson, what is it about this new technology that is so thrilling to you that you're gonna just, you know, rip up your current job and replace it with something a little bit more uncertain?
- 43:09
- Unidentified speaker: I don't think that case has been made to a lot of people.
- 43:11
- Unidentified speaker: I don't think so.
- 43:11
- Unidentified speaker: I tell my team, like, we are so lucky that we are a small, nimble, fast team that can just adopt every single tool, test it out, and then continue to compound on it.
- 43:19
- Unidentified speaker: When I walk by, you know, other offices and I see someone sitting on Microsoft Word, I feel bad.
- 43:25
- Unidentified speaker: I actually feel bad that they're living inside of these legacy enterprise tools where I talk to friends who are carrying two phones around because one phone can do all their personal stuff in their life and their kids, you know, you know, school calendars with Muse, but they cannot connect to their inboxes or their you know, Microsoft box folders for any type of you know cloud work.
- 43:45
- Unidentified speaker: People are literally living in two different worlds, it feels like.
- 43:47
- Unidentified speaker: Yeah.
- 43:48
- Unidentified speaker: It is definitely two different worlds, you know, and we haven't even talked at all about the moor doom and gloom uh prognostications from some of these places, which I think are not helpful, whether or not you think they're true, which uh, you know, we can have a long debate about that.
- 44:04
- Unidentified speaker: I do think that a lot of the way that some of these things have been communicated is effectively setting up a portion of the population to just kind of disengage from it because they're just like, either this is bullshit or it's gonna take my job.
- 44:18
- Unidentified speaker: Like, why do I need what is the point of me learning about?
- 44:21
- Unidentified speaker: Lose lose mentality for sure.
- 44:22
- Unidentified speaker: What is that?
- 44:23
- Unidentified speaker: Like, come on.
- 44:24
- Unidentified speaker: And it and I do think, you know, you can hold some of the leaders in this industry responsible for lack of quality communication, especially at the beginning, where it was like, yeah, it's probably gonna take 50% of all in entry-level jobs in a year.
- 44:37
- Unidentified speaker: Well, that's that's really tough to just state as a fact and then just kind of slide on to the next thing, you know?
- 44:45
- Unidentified speaker: I think it's hard.
- 44:46
- Unidentified speaker: And you want people to be more invested in their own change and their own career and what it means for them.
- 44:50
- Unidentified speaker: But I think it's kind of human nature that that hasn't happened in a lot of cases.
- 44:53
- Unidentified speaker: Yeah, we've done a poor job as an industry as well, showcasing the the benefits to the the common person, right?
- 44:58
- Unidentified speaker: You know, in healthcare and education and all the things that government's supposed to be doing better for us, you know, how this could actually do it faster, cheaper, better.
- 45:06
- Unidentified speaker: But I also think it's gonna empower people who are powerless as well, right?
- 45:10
- Unidentified speaker: Think about who in your life has an EA.
- 45:13
- Unidentified speaker: It's a perfect example.
- 45:15
- Unidentified speaker: You know, executive assistants were $50,000, $100,000 jobs on top of somebody's other day-to-day job that they would just pay to basically hand off stuff to.
- 45:24
- Unidentified speaker: That's an expensive thing to have an EA.
- 45:26
- Unidentified speaker: You kind of know you've made it into a senior position when you get an EA, right?
- 45:30
- Unidentified speaker: Think about like the madmen era.
- 45:32
- Unidentified speaker: Every single person now has the smartest EA in the world for free in their pocket.
- 45:37
- Unidentified speaker: Yes.
- 45:38
- Unidentified speaker: I don't understand what else you need to know about the benefits of this that is now at your fingertips.
- 45:43
- Unidentified speaker: You just have to accept that there are these huge benefits that were never available to you six to 12 months ago.
- 45:48
- Unidentified speaker: But we can go on this forever, Peter.
- 45:50
- Unidentified speaker: I don't know if there's anything else we missed.
- 45:52
- Unidentified speaker: I think that, you know, just to a point of empathy for the people who maybe haven't made this change yet.
- 45:57
- Unidentified speaker: I'll speak from my own experience.
- 45:59
- Unidentified speaker: There were ways that I was working, for instance, at Carta in the last few years where I had a hard time occasionally allowing AI to do the things that I felt I was great at.
- 46:09
- Unidentified speaker: Because I was like, what am I adding then?
- 46:11
- Unidentified speaker: Guy who lives in charts and graphs.
- 46:12
- Unidentified speaker: I was pretty freaking good at it for a while, to be honest.
- 46:15
- Unidentified speaker: And there was a little bit of grief to be like, actually, no, now the AI is gonna create this stuff for me.
- 46:21
- Unidentified speaker: And what I get to do is like think and decide where it acts.
- 46:25
- Unidentified speaker: That took a little while.
- 46:26
- Unidentified speaker: It wasn't that awesome to be like, hey, this thing that I spent 10 years getting really freaking good at, now it doesn't matter anymore.
- 46:32
- Unidentified speaker: And everyone can create great-looking charts and graphs, and I'm just kind of one of many.
- 46:36
- Unidentified speaker: You it did, it was like a little bit of a grieving process to get over that.
- 46:39
- Unidentified speaker: And then finding at the other side, oh, yeah, now I can spend my time in a hundred new and interesting ways alongside this super powerful colleague that can build charts or graphs better than I ever could.
- 46:49
- Unidentified speaker: So, what advice do you give to other friends, colleagues, or family members who are also not willing to accept that grieving period of not having to write in cursive or you know handwritten letters?
- 46:59
- Unidentified speaker: I mean, I think the the key point is that there needs to be an acceptance, whether it's in a day or a week or a month, of the inevitability of this change.
- 47:07
- Unidentified speaker: And then also coming along with that is the idea that the expertise that you built is not no longer valuable, but it has to be applied at the strategic rather than the tactical building layer of like the best thing that I can do now, having this experience that I've had in insights is to ask the right questions and provide the right framing.
- 47:27
- Unidentified speaker: And yeah, the AI is getting better at that too.
- 47:29
- Unidentified speaker: And I'm sure once that's taken up, I'll have to go find new things to do.
- 47:32
- Unidentified speaker: But I you just have to get comfortable with the idea that you need to find new things to do.
- 47:37
- Unidentified speaker: And that there are many, many, many new things to try, that some of which are just enabled by this technology rather than removed by it.
- 47:43
- Unidentified speaker: Yeah, it's such great advice.
- 47:45
- Unidentified speaker: You know, Peter, I can talk to you for days here, but but before we wrap things up, we always ask our guests for their fast favorites.
- 47:49
- Unidentified speaker: So, first off, your favorite new podcast or ones you've revisited.
- 47:53
- Unidentified speaker: Ooh, good question.
- 47:54
- Unidentified speaker: Favorite new podcast is probably the rest of his is history, which is probably an old podcast for many people, but I just recently started getting into it and it's awesome.
- 48:01
- Unidentified speaker: Nice.
- 48:01
- Unidentified speaker: How about your favorite newsletter or blog?
- 48:03
- Unidentified speaker: The graphic detail by The Economist is a can't miss if you're into charts and graphs.
- 48:07
- Unidentified speaker: Still, I love, love, always, love, love.
- 48:09
- Unidentified speaker: Always, always.
- 48:10
- Unidentified speaker: OG.
- 48:10
- Unidentified speaker: Nice.
- 48:11
- Unidentified speaker: How about your uh next is your favorite tech gadget?
- 48:14
- Unidentified speaker: Uh weirdly, I am not like the most gadgety person, but I did recently upgrade my AirPods, which are like four years, three years old.
- 48:21
- Unidentified speaker: And new AirPods are much better.
- 48:22
- Unidentified speaker: So if you're sitting around with old AirPods, definitely upgrade those.
- 48:25
- Unidentified speaker: You got to get the Apple care on the AirPods, and then when it expires in a year, you just go in and upgrade.
- 48:31
- Unidentified speaker: Yeah, the planned obsolescence on AirPods, I think, is actually very strong.
- 48:34
- Unidentified speaker: So that was a mistake on my part.
- 48:36
- Unidentified speaker: Okay, you learned your lesson.
- 48:38
- Unidentified speaker: How about your favorite new trend?
- 48:40
- Unidentified speaker: Oh, favorite new trend.
- 48:43
- Unidentified speaker: I'll I'll speak from uh place of bias here.
- 48:47
- Unidentified speaker: I love, love, love seeing all of these tech companies hiring economists or people who live in data and using Mac to promote their product.
- 48:55
- Unidentified speaker: Like, hell yeah, man.
- 48:56
- Unidentified speaker: More insights, more numbers, more like it's it's so cool to watch other, a lot of other companies like adopt this as something that's important because I just think it's a it makes for a much better broad conversation where you're not just reading product marketing all the time.
- 49:10
- Unidentified speaker: You can actually ground this stuff in evidence.
- 49:12
- Unidentified speaker: I mean, you were definitely a trailblazer in that.
- 49:14
- Unidentified speaker: Don't uh take offense to this, but you're like the OG and tech head of insight and economist stuff.
- 49:20
- Unidentified speaker: Come on.
- 49:20
- Unidentified speaker: I built on a lot of other people's work.
- 49:23
- Unidentified speaker: How about next is your favorite book or new book?
- 49:25
- Unidentified speaker: I just finished Kevin Roos's AGI Age, which is going through the background on Demi and Dario and Sam.
- 49:32
- Unidentified speaker: Uh a little pop, you know, psychology in that, but I found it really interesting.
- 49:37
- Unidentified speaker: Awesome.
- 49:37
- Unidentified speaker: And last but not least, your favorite life lesson.
- 49:40
- Unidentified speaker: Let go of your toys.
- 49:41
- Unidentified speaker: Let other people play with them, even if those other people happen to be agents.
- 49:45
- Unidentified speaker: Let go.
- 49:46
- Unidentified speaker: Grie for a little and then move on to greener pastures.
- 49:49
- Unidentified speaker: Thanks again for jumping the tank today, Peter.
- 49:51
- Unidentified speaker: Appreciate you, Matt.
- 49:52
- Unidentified speaker: Hey everyone, thanks for tuning in to another episode of Tank Talks.
- 49:55
- Unidentified speaker: We hope you found today's conversation as insightful as we did.
- 49:58
- Unidentified speaker: If you're enjoying the show, we've got three quick things to ask of you.
- 50:01
- Unidentified speaker: First, hit that subscribe button on your favorite podcast platform so you never miss an episode, whether that's Apple Podcasts, Spotify, Google Podcasts, or YouTube.
- 50:11
- Unidentified speaker: Next, follow us and stay up to date on upcoming episodes and behind-the-scenes content on social media with Twitter, LinkedIn, and Instagram.
- 50:19
- Unidentified speaker: And lastly, share the love.
- 50:21
- Unidentified speaker: If you found value in today's episode, share with a friend or colleague who benefit too.
- 50:25
- Unidentified speaker: Your support helps us bring in more amazing guests and keeps the Tank Tots engine running.
- 50:30
- Unidentified speaker: That's it for today.
- 50:31
- Unidentified speaker: Until next time, keep disrupting and innovating