November 9, 2026 · Variant · New York
Inference is Crypto
Playback could not load. Open the recording directly.
How pricing, routing, and provider incentives shape AI inference markets—and why verification matters.
Transcript
Machine-generated transcript, formatted into paragraphs for readability. Names and technical terms may contain errors. Sentence timing is approximate; select a sentence to play from there.
Inference is Crypto
0:00Hey, Ron. So my title is really Ragebait. This is a talk about inference, but it's more about there's a lot of analogies in the inference market to crypto, and a question is, are people exploiting those? I would say, you know, luckily I'm giving this talk today where at Fireworks, one of the largest inference writers was exploited, and people did something kind of very related to what we're going to talk about here using the hacked compute capacity.
So, you know, timely in some ways. But the main thing we're really going to talk about is when you use a model and you're not running it locally, how much can you really trust what the provider is giving you? And how can you tell when they're manipulating you?
The growth of open-weight models
0:49So, you know, I think probably most people in the room are familiar with sort of open-weight models as sort of the counter to Frontier Labs. You know, the history of them is pretty short, but you can already see the parameter count of these models growing dramatically. There have been a bunch of marketplaces that have been built around running open-weight models for you because as they got really large, the dream of the local model is kind of dead.
And we'll talk about it a little bit. But if you actually look on some of these marketplaces, which allow you to either use open tokens or closed tokens, you actually often find that the routers that they're choosing where your tokens are going are picking open models a lot more because they're usually cheaper. We'll talk about what model routing really does. But all in all, if you kind of look at all the metrics, open tokens are going up exponentially, potentially faster than closed source models.
Why inference moved to providers
1:56But no one is really running these locally anymore. So to give you an idea of just like how large these are getting versus, you know, the dream of the local model you run on your laptop, you know, the original 2023 llama, 65 gigs, most people in this room probably have a laptop that you could fit that on locally. But as you can see, like once we got to sort of the larger DeepSeek v4s and then of course Kimi K3, which is sort of the largest well used open source model, even having eight H100s, which are reselling above list price.
So that's, you know, 80k roughly of hardware. You can't really run a lot of these models. And so that's created a whole business around people who are bidding on your tokens, like bidding for your orders to process for you. So, you know, for instance, these are just numbers from OpenRouter on GLM 5.2, which is a ZAI model. An interesting thing to note is that the people who built the model, who trained the model, who spent tens of millions of dollars investing in making the model work, it turns out that they don't run their own model the fastest.
And in these competitive marketplaces, there are people who are offering you a price per token and also some other properties like latency, how much, how many sub agents you can run simultaneously and other things. And so there's kind of this interesting phenomena, which is a little bit like crypto, where you have open source software. The people who wrote the open source software are amazing at maintaining that, but are absolutely shitty at running it in production.
And there are third parties who take it and optimize it and do a lot better than them. And that's created a kind of novel market structure here.
Unbundling the AI stack
3:48You can sort of think of these open weight models as taking Claude, taking OpenAI and unbundling it. And what do I mean by unbundling it is, you know, when you use these closed source services, you're using a front end that is provided by them. You're using a model router, which is choosing whether to use, you know, 6.1 Sol or use Fable for something or use whatever. And then you're also sort of using the model itself, of course.
And then finally someone is running the GPU somewhere or the CPUs and GPUs for you somewhere. But you're buying the whole bundle, right? You're paying the subscription and you're getting sort of all four of these services. But if you look at the open weight marketplaces, the open weight marketplaces have many different components with different competitors who are trying to optimize each layer separately. And so in this sense, you get this sort of distributed optimization where individual teams are working on separate components and optimizing them.
And then you as a user put them all together and then generate your own version of Claude using an open source model. So that's sort of how you should, from an economic standpoint, I start with this thing that's bundled. I'm unbundling into these components. And then people are trying to sell me each of those components and then I try to make the same experience I have with the closed source model.
And you might say, why do this? Well, it's 90% cheaper. So that's in many cases the real reason.
Your prompts are order flow
5:19And so in crypto, it's you sign a trade or transaction, you send it to your wallet, your wallet oftentimes will route you through an aggregator that tries to find you the best price to execute transaction. And then the stuff Dmarz just talked about where people are front running you and stuff like that. That's this thing that's called MEV. So if you believe that this market structure is the same as the open source market structure, a natural question is do these kind of malicious or like consumer welfare hurting actions also happen in the interest market?
So that was my question in April of this year. I started basically scraping a lot of OpenRouter data because like it's a little bit like a blockchain that it's all for the most part public. And so the question is like from observing that for a few months, can you find examples of providers doing some malicious act, front running you, giving you the wrong tokens, quantizing, dot, dot, dot.
There's not a totally new thing. I think there's actually a lot of papers at ICLR and NeurIPS last year on model providers and gateways over quantizing. So giving you a shittier quantization and then how do you detect that and then it turns out you have to spend more money to detect it than it's worth you paying for. So that part is again also a little bit interesting.
How OpenRouter chooses a provider
7:07So, you know, let's I'm going to kind of distill what OpenRouter does at a high level. Obviously, there's many AI gateways now. So for cell OpenRouter ramp Palantir Databricks, basically anyone who has a pulse and owns users data. So in the open world, you will have your harness or front end. Here I gave an example of Hermes. Your front end will oftentimes choose a model for you.
Maybe it's like you say, hey, I want to use GLM because I love ZAI and I hate minimax. Or maybe it will try to figure out which model is best for your task. And so that's called capability routing. So we're not really going to talk in this talk about capability routing. But at a very high level, the way it works is I take my prompt. I take your prompt.
I go look at a bunch of evals that are public or maybe private that the router is generated from its user data. And I say, what evals does this task look like? Okay, similarity score 10% to this eval. Similarity score 20% to this eval. And then from the evals, I say, okay, how much does it cost? How much do I think it will cost to run this task?
And how much are people quoting the price for and trying to look at utility minus cost? And then choose a model from that objective function. So that's sort of what capability routing is. Then we get to execution routing. And this is where it sort of looks like crypto. There's people doing real-time bidding of how much they're charging you for an input token. The platform, OpenRouter, gives you statistics of what's their time to first token.
So that's like latency to first token for audio, video, and text. And then you can see their historical share of traffic. Like how much token order flow did they get? Also, when I say token here, I mean AI token, not crypto token. I know it's kind of confusing, unfortunately. And then the platform will also kick out providers who are not doing a good job. So maybe they might have quoted a really cheap price, but they over-quantized or they don't give you the tokens you wanted or whatever.
Repricing and the Tullock contest
9:20And so how does it work? So there is a thing called a Tullock contest. A Tullock contest is a sort of procurement auction for work where people bid in a price. And then the auctioneer chooses them probabilistically according to some function of their price. So in the case of OpenRouter, for reasons unknown to me, but when I asked Alex, the founder of OpenRouter, he was just like, yeah, it sort of looks like Uniswap.
And so that's, again, their choice of this Tullock contest is also kind of came from crypto, ironically. So basically, that means a long-running average you should see is that the market share you get is sort of quotable by like one over price squared. And so it says R equals two in the docs. Obviously, there's some filtering, right? They're kicking out some providers. So some of those providers provide price, but then their thing gets probability zero.
So it slightly changes it. So empirical evidence, and if you want, you can read the whole post for the methodology, but it's, you know, pretty close exponent-wise set. So it's pretty clear they are doing something like this. And what's happening here is each of them is quoting a price, and you can see kind of how much market share they're getting based on how they're quoting.
Why agents make caching matter
10:42So agentic tooling kind of makes how you price very complicated, but also kind of unique to this marketplace. So generally speaking, the OpenAI API, when it first came out, offered you three diagnostic statistics on a prompt. How many input tokens were used? So how many tokens from the prompt that was prefilled or used? So prefill is the taking the input tokens, figuring out all the attention vectors, and caching them effectively.
How many were cached? So how many times am I rereading the same token? I already know the matrix vector product. I don't need to recompute it. And then finally, how much are outputs? So outputs the most expensive because it's not, and oftentimes this will be called decode. Decode is the most expensive because A, I have to run the full model, but then B, I also have to stochastically sample from the distribution of tokens, and maybe I have to sample top K and then choose K, right?
There's a lot of nuanced tricks into how you sample output tokens, and this is actually why a lot of the people who have become really good at optimizing models for serving, you know, how I was talking about people who train the models oftentimes are shitty at serving. Like their empirical metrics like latency, bandwidth, etc. are very crappy compared to these other providers. Because these other providers have spent all their time figuring out how to optimize sampling the next token and doing the sampling process.
They treat the model as fixed and focus on this. And so I think that's where you're seeing a lot of this. So why do I bring this up? Well, agentic task, agentic coding reuses a lot of your context, right? It's like a coding task, you say, like, make me a website. Okay, it writes like the first thing goes to Python, import some libraries, writes the first version.
That gets appended to the context and resent. That means those tokens are going to get reread a lot because it's resending a lot of the same prefix over and over and over. And so that's kind of what this thing to the left is showing you is this idea that actually you have a lot of cache tokens, right? And I think this is actually what has made the coding models so much better is the fact that people optimize this step.
The repricer playbook: do not cache
12:53So there's a notion of someone who is trying to manipulate this that's very simple. This is like the stupidest strategy you could think of in a relatively complicated market, which is actually I'm just going to say I'm never caching. Because if you look, there's the cache tokens, I charge you a lot less than the other two. So why not just lie and say I'm just never caching?
Because it's pretty easy. There's no verifiability. I have no cryptographic signature of how you structured your cache or how you did pre-fill. And so I could just keep returning zero. And then I can charge the user more. So the question is, does this exist in this data? A very simple kind of analogy here is I make my price per input token really low. So the router gives me more tokens.
But then I say I never cache. So then after many steps of an agentic workflow, I will actually end up, the user will end up paying me more than they would have paid the person who's honestly caching.
Half the price, twice the bill
13:54And so you can think of this as the person who, you know, kind of schematically, this person says I charge $1 per input token, $10 per cash token. And this is how much your costs will be after 10 steps. This other person who lies and says they are never caching, even though for themselves they would definitely just use the cache. Because otherwise they are spending more money running the hardware.
This is how much money they would make after 10 steps. So hopefully very simple. There is no complicated math here for that.
The same session, turn by turn
14:25This is just sort of to give an explicit example so you can see, like, okay, this is where the turnsquad is. This is just the table of the previous numbers.
Evidence from OpenRouter
14:34So you might say, hey, is this happening? And the answer is yes. I found a lot of evidence that there were a small number of entrance riders, interestingly enough, 50% of them were all Ethereum miners before. But who were doing exactly this, where they were pricing very low on input tokens, taking all the order flow, but then saying they never cache. And to figure this out, I basically spent $5,000 or $10,000, I forget, on running experiments every day and trying to look at the sample.
Like, was I getting cash? Like, I had to spend money to learn this fact. It was not like a free fact you can divine. Which, again, kind of gets to the nature of this type of thing. It's like, yes, I can verify it, but I have to spend a lot of money to actually verify it.
Pricing in the cache
15:21So OpenRouter released a new router October 28th, so it's pretty fresh. And one thing it started doing was including cash in the routing quality. So how good your historical cache hit rate is shows up. And another thing is they do sticky sessions, which means one interesting thing about the routing and this bidding process is every time I switch providers, I blow the cache, right? Because, like, the new provider was not running the previous context.
They have to go do the pre-fill. They have to set up the decodes. And so they started adding a lot more in terms of sticky sessions. And so in general, this does reduce the amount of these types of attacks, but this is only one attack. This strategy space is much larger than this.
Routing as an ad auction
16:11So I think the correct analogy for you to have of, like, what the new router looks like without going into details is it's kind of like an ad auction. Right? So in ad auctions, when you bid, you're usually not just bidding in price. You're bidding in quality also. Like, how much of a click-through rate do I get per dollar? Right? It's not just I'm bidding this amount.
And so the new OpenRouter model is basically the amount you price, sort of the expected mix of the three tokens. So this is where they use your historical cache hit rate divided by the success rate. So they're making some model prediction on each provider that is a quality score effectively and that quality score times price. So, and it's actually very similar to some classical online ad auction, the online ad auction literature.
But the interesting thing is the ad auction literature suggests there's a ton of other ways you can manipulate this system, which I'm trying to do some experiments on now. But there's, I don't want you to get this idea that, oh, like, oh, if you fix caching, all of a sudden there's, like, no other attacks. There's a million other attacks.
More tokens, more ways to cheat
17:18And as I was saying, there's a bunch of different tests people have done where, like, all of these, what I've done, what they've done, we have to spend money to figure out that this is happening. And the question is, like, how do you minimize the cost of tests you have to do to verify that such actions happen with high enough accuracy? And the more tokens there are, so this is sort of the growth curve of OpenRouter over the last year.
The more tokens there are, the more some strategies that were not profitable will suddenly become profitable. And that sense is very much like crypto.
Do not trust, verify
17:52So, you know, I guess I will end with the natural crypto phrase of don't trust, verify. Although it's expensive to verify, unfortunately. But the three lessons are, you know, score what users actually pay, try to build a quality score, and if it was possible, getting, you know, some sort of cryptographic signature for the cache tokens would work. And you can read the post for all the technology.
Q&A: trust, incentives, and order flow
18:23And I want to move on to Jude. I thought this was a really interesting comparison. It feels like where the comparison breaks down to me is that in crypto, by definition, everything is adversarial. No one's cheating. They're just following the protocol. Whereas in these networks, it feels like people are cheating. and so it seemed to me in the inference case, there's lots of layers in the stack like people can build a brand and say like, maybe OpenRouter is gonna say like, we're gonna spend money to investigate these people, that sort of thing.
Like it feels like the incentives are different and so I would expect like a different outcome but I'm curious if you see it that way. I think there is some truth to that but I actually think the strategy space is way larger than crypto. In crypto, because assuming you can't break the cryptography, I'm giving you constraints and putting you in a box on what sets of things you can do are.
Here, it's like, ah, it's fully open. And I don't know enough about the workflows ahead of time to fully do this. Now, I think the routers are policing. So, for instance, OpenRouter just added US inference providers only today, which they charge you 3% more for, which is kind of hilarious. The second thing is that there's a ton of routers now. And it's a little like crypto exchanges.
Some will be like KYC friendly, whatever. Some will be like do whatever. And so this fireworks attack that we're talking about today, basically the attackers hacked fireworks, took a bunch of their B300s and started reselling them on these routers that had permissionless entry and exit, which is sort of like your non-KYC crypto exchange. So I think the thing is the router, the open source thing, everyone can copy each of those components that I broke things down into and go create a new market out of it.
So I'm not totally sure it will be that different, is kind of what I'd say. One more. All right, we'll go with Denison. So persistent. Yeah, so I run an ad sponsored harness. and one thing that I've seen with the less reputable routers is they're subsidizing the cost of inference through distillation and they are somehow like classifying the value of your traffic ahead of time and then if it's not valuable traffic they just drop it they won't serve you and then if it is valuable traffic then they will serve you so they have this weird reliability where you're sort of like putting up with it for price but they give it to based on how valuable the task is that you're solving, or they'll batch it, which is really low quality in the responses.
But it's really fascinating to see all the kind of ways that people are trying to cheat one another. For sure. I mean, an interesting thing in OpenRouter you've already seen is some of the inference providers who are model developers themselves, so thinking machines in particular, offer two different endpoints in OpenRouter. One that is ZDR, but of course it's all trust me, bro, ZDR, meaning zero data retention, that will charge you a higher price, and then one that's like, you allow us to use your traces, whatever, which I think the harnesses actually have the best position to be kind of aggregating and selling, which is sort of what's happening there.
I think another interesting thing is, as a disclaimer, also I've invested in a lot of harnesses and inference writers, but the inference writers are already paying for order flow, like exactly kind of like crypto where like the harnesses are like, okay, we have a hundred billion tokens a day. The inference writer will come to you and be like, we'll give you a rebate of X dollars per day.
If you could guarantee us one billion tokens a day for the next two months. Right. So you're already kind of seeing a lot of alternative monetization. Let's put that way. It's pretty, it's pretty fun. All right. Well, thank you to room. You can find them afterwards. I'm sure you have lots of interesting answers. Okay, so our next...
Slides
Continue exploring















