When an agent gets expensive or slow, four ideas come up. They sound alike and they are often mixed up. They cut different things.
Prompt caching
Your provider reuses the part of the prompt that repeats from call to call, usually the system prompt and tool definitions. Each call costs less and starts sooner. Every call is still made.
Model routing
Easy steps go to a smaller, cheaper model and hard ones to a larger model. Each call costs less. Every call is still made, and the smaller model has to be good enough for the steps it gets.
A semantic cache
When a question comes in that looks like an earlier one, the earlier answer is replayed. That removes calls, and it works for answers that don't change. For an agent that acts, it breaks down: a new customer and a new order need a new action, not an old one replayed.
Compiling the repeated jobs
AgentCompile learns the jobs your agent repeats from your agent's own history and runs them without calling its model. It isn't a replay: the job is learned, so new customers, new orders and new details are handled. Anything new, unclear or unusual goes to your agent, unchanged.
A cache replays answers. A compiled job does the work again, without the model.
Which ones work together
- Prompt caching and model routing make the calls you still make cheaper. Keep them.
- Compiling removes the calls the repeated jobs would have made. It wraps the client you already use, so those two keep working.
- A semantic cache and compiling aim at the same calls. For agents that change things, compiling is the one that handles new details.
Questions we get
- Is AgentCompile a cache?
- No. A cache replays old answers. AgentCompile learns the job, so it handles new customers, new orders and new details.
- Is it fine-tuning?
- No. Your model stays exactly as it is. AgentCompile decides when a call to it isn't needed.
- What happens to calls it doesn't handle?
- They go to your model, unchanged, with your own key, and prompt caching or routing apply to them as before.
What the four look like in an agent's own logs. Read: What your agent does all day