An AI agent doesn't call its model once per conversation. It calls it on every turn of its loop: read the message, decide on a tool, read the result, decide again, reply. Each of those calls reads the whole conversation so far. That is why agents cost more and take longer than the chat demo that inspired them.
There are several ways to bring that down. They are often lumped together, but they cut different things. Some make each call cheaper. Only a few make calls go away.
Where the calls come from
- Every step of the loop: each tool decision and each reply is a model call.
- Reading tool results: after a lookup, the model is called again to read what came back.
- Retries: a malformed tool call or a timeout means the same step runs again.
- Repetition: the same job, done for a different customer, is reasoned out from scratch every time.
Five levers, and what each one cuts
- Trim the context. Send each call less: shorter instructions, fewer old turns. Each call gets cheaper; the number of calls stays the same.
- Turn on prompt caching. Providers can reuse a repeated prefix, like your system prompt, across calls. Each call gets cheaper; the number of calls stays the same.
- Route to a smaller model. Send easy steps to a cheaper model. Each call gets cheaper; the number of calls stays the same.
- Add a semantic cache. Replay an earlier answer when a similar question comes back. Calls go away, but an agent that changes things can't safely replay an old action for a new order.
- Compile the repeated jobs. Jobs your agent does again and again run without calling its model, with the new details filled in. Calls go away, and anything new still goes to your agent.
Most levers make each call cheaper. Only a few make calls go away.
What compiling did on a benchmark
On τ-bench retail, the public customer-service agent benchmark, with simulated customers, compiling the repeated jobs meant 58% fewer agent calls: 17.2 to 7.3 per conversation, with the same answers. None of it is customer data. A pilot measures your own traffic.
Where to start
- Count model calls per conversation, not per day. That is the number every lever moves.
- Read your logs for jobs that repeat. Most production agents spend most of their calls on a short list of them.
- Trim context and turn on prompt caching. Both are cheap to do and keep working alongside everything else.
- Take the repeated jobs off the model's plate, and keep your agent as the fallback for everything else.
Questions we get
- Does prompt caching reduce the number of calls?
- No. It makes the repeated part of each call cheaper. Your agent still calls the model on every turn.
- Is model routing the same as compiling?
- No. Routing still calls a model, just a smaller one. A compiled job doesn't call your agent's model at all.
- Can these be combined?
- Yes. AgentCompile wraps the client your agent already uses, so prompt caching and your choice of model keep working for every call that still goes to your model.
Add AgentCompile to your agent in one line. Open the Cookbook