IQ Routing
Trajectory-aware LLM routing that cuts agent cost
81 followers
Trajectory-aware LLM routing that cuts agent cost
81 followers
IQ Routing is a drop-in gateway. We classify each request, serve from cache, and route to the cheapest model that clears the quality bar and completes the task. The insight others miss: an agent is a trajectory, not a stream of independent calls. We route each step based on where it sits in the run: a cheap model on boilerplate, the strongest model on what matters.
Does the cache reuse cross runs? That is the piece I can't place under your own trajectory framing. If it does, a step gets handed an answer produced under a different set of earlier steps, so a request that reads identically can sit at a very different point in the run.
Haven't tried it yet so this may well be handled. It's the first thing I'd look for in the docs.
@dimhold The cache key for any request with prior conversation turns is built from the full history up to that point, not just the current message. A cache hit only occurs when the entire trajectory leading into that step matches as well, so a change anywhere earlier in the run changes the key and produces a miss instead of a hit. That’s the mechanism that rules out a step being served an answer computed under a different set of earlier steps.
The one exception is the first call of a run, before any history exists to hash. That request is matched on content alone. Since there’s no prior-step context at that point, it falls outside the scenario you’re describing.
@george_avila That closes it, thanks. Full-history keying was the part I could not see from outside.
The first call is the one I would still watch. My own runs open with a byte-identical first message every time, same system prompt and same task template, while the repo underneath has moved. A content-only match hits there. What comes back was computed against a different working tree.
I have not run IQ, so I may be reading that exception too broadly. Do you scope the first-call key by anything outside the message?
@dimhold A cached response is matched only on the content of the request itself, meaning the instructions and the message text, rather than on anything about your environment or repository state. Given that, an identical opening message can return a cached response even after what’s actually on disk has changed since that response was generated, and the same holds for near-duplicate opening messages, not just exact ones.
To limit that exposure, cached responses expire on a schedule rather than staying valid indefinitely, and responses involving tool use are given a much shorter shelf life than plain text responses, since they typically depend more on something outside the request itself. That narrows the exposure window, but it does not close it entirely.
If a live call is ever required for a specific request, sending a request header (x-iqr-cache: off) opts that call out of caching entirely, and nothing else about it changes.
@george_avila That is the answer I was after, thanks. The shorter shelf life for responses that involve tool use is the part I would not have guessed.
For my case the header is the simple fix: set x-iqr-cache: off on the first call of a run and leave everything after it cached. The opening message is the one place where identical text does not mean identical world.
Hey George — was reading through the security docs and noticed that with payload logging enabled, retained payloads can be eligible for the manual training export, but there doesn’t seem to be a way to keep payloads for debugging while marking them as no-training.
Has that trade-off come up with customers yet, especially enterprise teams with stricter data policies?
@viktor_arakelyan Right now, payload retention is an org-level switch, not a per-request one. You turn it on for debugging, with an expiry window, or you leave it off. When it's on, that same data is what the manual training export can draw from. There's no separate flag yet for "keep this one for debugging, never train on it." We do have a Zero Data Retention setting that locks capture off permanently for orgs that want a hard guarantee, but that trades away debug visibility instead of solving what you're pointing at. Still early on the customer side, so I don't have a live example to point to yet, but it's a fair thing to expect from teams with stricter data policies, and worth having an answer for before it's asked.