Spend Less Tokens

You can cut your token spend by over 90%. 

A year ago, token spend wasn’t even part of the conversation. I remember folks telling me that ‘the technology was going to transform their business’, so they weren’t all that concerned about what they were spending on tokens. Nowadays it’s easy to make fun of token-maxxing, but at the time this was rampant. 

Fast forward to today, and this has changed quite a bit. AI usage within large companies has gone up dramatically, with token spend alongside it. Productivity has gone up only marginally, and it certainly hasn’t translated into the financial returns that most executives hoped for. Now, they want to know why the bill keeps growing, what they should be measuring, and how to bring down the cost of agents without reducing the efficiency gains.

At Varick, we build agents for the largest companies on the planet. Our core focus is proving the largest possible efficiency gains for the cheapest possible token spend, and as a result, we’ve noticed the same patterns surfacing over and over again. 

Daniel, who runs our FDE team, recently wrote an article about how companies will soon manage token spend the way a trading firm manages capital: moving dollars away from lower-return use cases and reallocating them towards higher-ROI ones. This is more of the financial side. But what about the engineering side? 

If intelligence is expensive, how do you use as little intelligence as possible, while making sure the intelligence you do use is doing useful work for as cheap as possible? Through this framing, measurements become clear: you ought to track ROI on intelligence, not raw spend, or even spend per employee. 

Put simply: if $10 of Fable 5 solves 1 hard task, and $10 of Gemini 3.6 Flash solves 100 easy ones, the question becomes how many of your “hard” tasks are actually just bundles of “easy” tasks. Spoiler alert: it’s most of them. 

Once you break down a complex workflow into individual steps, you realize most steps when calling models don’t need SOTA intelligence at all. Some steps can be deterministic, some can use smaller models, and some outputs can be cached instead of generated again every single time.

90% of your company's token spend never needed to happen in the first place. Spend less tokens.

Most enterprise workflows are deterministic

One thing we started noticing pretty quickly once we went deeper on token spend was that a lot of the workflows people called agentic only needed LLM judgment in a few places, yet teams were using AI for all of it.

From the outside, these workflows looked very complicated (writing to multiple systems, different paths the workflow could take, approvals/exceptions). We would look at the complex workflow and determine that we’d need a sophisticated agent with frontier AI to run it. 

But once we started going through these workflows step by step, we could see that a lot of the work the "agent" was doing was simple and deterministic: checking whether fields matched, moving data from one place to another, or just applying a rule. We realized these trivial tasks can be done with code, which is much cheaper, and we were wasting tokens by running them with expensive intelligence.

What we’ve come to learn is that the number of steps in a workflow and the amount of intelligence required in that workflow are two very different things. 

Once you separate them out, a lot of enterprise workflows are mostly deterministic, with only a very small number of places where the model really needs to think. Anything deterministic can and should be done in code, meaning the AI in most of these workflows was overkill.

The workflow is the wrong unit of automation

Once we started looking at workflows this way, we realized we had been thinking about the wrong unit of automation.

Models are now good enough where you can give them a lot of context and access to a few tools, and let them work through a pretty complicated process on their own. But that also means the model ends up being used in many areas where it wasn’t really necessary, and blows through your token budget for seemingly no reason.

That's the biggest design mistake we see: automating at the workflow level. Break the workflow down into the individual tasks inside it instead, because each task has a completely different intelligence requirement.

Once you know exactly which workflow components require AI and which don’t, you can be a lot more granular about where AI gets used, and you end up spending tokens only in the places that require judgment.

These are mainly the parts of the workflow where you can't know the right answer ahead of time because it depends on the situation. For example, imagine an agent gets an email. That email might mean different things depending on what the customer was asking. Whenever the answer depends on background context and there's more than one plausible way to handle it, that's where a model becomes useful, because there's an actual judgment call to make.

But the next mistake we see is pointing the heaviest frontier model at every task that requires a little bit of judgment. Our methodology is to break the workflow into individual tasks, then benchmark models against each one and use the smallest model that reliably handles it, instead of handing the whole thing to a frontier model in one call. By the end, we have a system that runs Gemini 3.6 Flash where possible and only reaches for Opus where absolutely necessary. The rough distribution is 90% non-frontier, 9% near-frontier, and 1% frontier. We call this the 90/9/1 split. Priced against running everything on frontier, that split is where the 90% at the top of this article comes from.

The model should become a smaller part of the system

Once you assess how much intelligence a step really needs, you begin to realize most steps in a workflow don’t need a powerful model. In fact, some steps don’t need a model at all. 

Deterministic steps should be plain code (e.g. if we get an email, we should download its attachments, if it has any). Very plainly: If X, then Y. 

And simpler reasoning should go to smaller, cheaper models, like Gemini 3.6 Flash (e.g. read the line item on an invoice and if it’s a stapler, call an LLM to GL Code it to the right category - in this case ‘office supplies’).

And known outputs can be reused again from cache instead of generated net-new every single time (e.g. every time you read a line item, you should map its GL Code to the cache, so that whenever you see ‘stapler’ you know it’s ‘office supplies’ without even calling the LLM). 

Make every model call earn its place

Start with the workflow as it actually runs today. You’ll usually find many places where the outcome can be determined without a model. Those should just become code. If this, then that. You know, the good old-fashioned way. 

The remainder will be the small areas where judgment is required. Models are necessary there, but the steps don't all need the same amount of intelligence. A smaller model might be completely fine for one step, while another needs something stronger. With these steps, keep a human in the loop when there's a lot of judgment or domain knowledge required, or when the answer is too costly to get wrong.

Context is another big area where tokens are being inefficiently spent. The past 2 years of building in applied AI have taught us that the harness around the model has a very big impact on the amount of tokens spent, and my cofounder Eyad wrote an article about how to build one properly a few months ago. I'd highly recommend sending that article to whoever builds your agents.

A good harness prevents the model from being inundated with information that it doesn’t need. If the decision comes down to just a few fields and maybe a single document, there’s no reason to send the entire history of a workflow per LLM call. Narrow context is also more accurate, since irrelevant information is exactly where models get confused.

TLDR

• Many workflows that look agentic from the outside are majority-deterministic once you break them down. 

• You should rarely, if ever, let one model reason through an entire workflow, as opposed to breaking a workflow down into individual tasks, and then deciding task by task what actually needs a model.

• For each task, inspect deeply the amount of judgment that is actually required. If the answer can be known ahead of time, use code. If some judgment is required, use the smallest model that can handle it reliably. If the downside of being wrong is high, escalate to a stronger model or a person.

• Keep context as scoped down as possible, and reuse outputs when the same inputs keep showing up via caching. 

• Long story short: the goal is to spend intelligence sparingly, not just for token spend, but for accuracy as well. 

AI not working the way you had hoped?

This is exactly what we do for clients at Varick. Most of the work is no longer proving that an agent can automate a workflow. It is designing the workflow so that it makes economic sense when you run it millions of times. And doubly, ensuring that the work is actually being taken off your team's plate: reliably and 24/7. 

If you're a large company trying to get a hold on AI, that’s the problem we solve. Book some time to chat with our team of AI experts on our website today. We only have a few engagement slots left this year - varickagents.com.