Cache
Never pay for the same answer twice.
When a request you have served before comes in again, Cache replays the stored answer and skips the model entirely, so the repeat is near-instant and costs almost nothing. Exact matching is automatic, semantic matching catches paraphrases, and a hit short-circuits the whole pipeline.
exact · semantic · instant
A hit skips the model, the firewall and retrieval.
Your traffic repeats. You pay full price every time.
The same questions arrive again and again, often word for word, and each one runs the full pipeline and bills for a fresh model call. You are paying full latency and full cost to recompute answers you already produced minutes ago. A do-it-yourself cache then has to decide what is safe to store and when it goes stale.
How it works
Match, then short-circuit.
Turn it on and caching is automatic. The cache keys on the request as you sent it, before any other service runs.
- 01ExactMatch exactlyA key is built from the request as sent: the model, the messages, the sampling parameters and any tools. An identical request is served the stored response.
- 02SemanticMatch a paraphraseOn an exact miss, the platform embeds the prompt and looks for a near-duplicate within a similarity threshold you set, so paraphrases hit too. Raise the threshold for tighter matches.
- 03SkipShort-circuit the pipelineA hit returns immediately, before retrieval, memory, the firewall or the model run. That is what makes a hit both cheap and fast.
Four settings, and the last one bounds how stale an answer can get.
Why it is different
A cache that knows when not to.
A blunt cache returns stale or wrong answers and erodes trust. This one is careful about what it stores, how long it keeps it, and what a hit skips.
Reproducible only, by default
Temperature 0 or a seed. A request that asked for variety gets a fresh answer every time, unless you explicitly opt in to caching volatile responses.
Exact is exact
Exact matching never returns a wrong answer. Semantic matching is optional and threshold-tuned, so you trade precision for hit rate only when you choose to.
Bounded and clearable
A TTL bounds how stale an answer can get, and Clear cache regenerates everything at once when a prompt template, corpus or model changes upstream.
Honest about the short-circuit
A hit skips the firewall and the router because the stored answer already passed them. New policy applies to fresh requests, or after you clear the cache.
Fast and cheap
Without the footguns.
Deterministic by default
Only reproducible requests are cached, those with temperature 0 or a fixed seed. Volatile requests pass through untouched, unless you opt in to caching them.
Isolated and bounded
Your cache is scoped to your organization, never shared across tenants, and bounded by a TTL you set from 1 hour to 30 days. Clear it whenever you want.
Cheaper and faster
A hit records a small cache charge and skips the inference charge. That skipped charge is your saving, and the answer comes back in milliseconds.
FAQ
Questions people ask about Cache.
- Will it ever return a wrong answer?
- Exact matching cannot: the request has to be identical. Semantic matching is optional, and you set the similarity threshold at which a paraphrase counts.
- What gets cached?
- Only reproducible requests by default, meaning temperature 0 or a fixed seed. A request that asked for variety passes through untouched unless you opt in.
- How stale can an answer get?
- As stale as the TTL you set, from 1 hour to 30 days. Clear cache regenerates everything at once when something upstream changes.
- Is our cache shared with anyone?
- No. It is scoped to your organization and never shared across tenants.
- Does a hit skip the safety checks?
- Yes, and deliberately: the stored answer already passed them. New policy applies to fresh requests, or after you clear the cache.
Start here
Stop recomputing the same answers.
Create a key, turn on Cache, and serve your repeat traffic from a stored answer in milliseconds.