# Cache: never pay for the same answer twice Replays a stored answer for a request you have served before and skips the model, so the repeat is near-instant and costs almost nothing. ## Never pay for the same answer twice. When a request you have served before comes in again, Cache replays the stored answer and skips the model entirely, so the repeat is near-instant and costs almost nothing. Exact matching is automatic, semantic matching catches paraphrases, and a hit short-circuits the whole pipeline. exact · semantic · instant A hit skips the model, the firewall and retrieval. ## Your traffic repeats. You pay full price every time. The same questions arrive again and again, often word for word, and each one runs the full pipeline and bills for a fresh model call. You are paying full latency and full cost to recompute answers you already produced minutes ago. A do-it-yourself cache then has to decide what is safe to store and when it goes stale. ## Match, then short-circuit. Turn it on and caching is automatic. The cache keys on the request as you sent it, before any other service runs. 1. Match exactly: A key is built from the request as sent: the model, the messages, the sampling parameters and any tools. An identical request is served the stored response. 2. Match a paraphrase: On an exact miss, the platform embeds the prompt and looks for a near-duplicate within a similarity threshold you set, so paraphrases hit too. Raise the threshold for tighter matches. 3. Short-circuit the pipeline: A hit returns immediately, before retrieval, memory, the firewall or the model run. That is what makes a hit both cheap and fast. ## A cache that knows when not to. A blunt cache returns stale or wrong answers and erodes trust. This one is careful about what it stores, how long it keeps it, and what a hit skips. ### Reproducible only, by default Temperature 0 or a seed. A request that asked for variety gets a fresh answer every time, unless you explicitly opt in to caching volatile responses. ### Exact is exact Exact matching never returns a wrong answer. Semantic matching is optional and threshold-tuned, so you trade precision for hit rate only when you choose to. ### Bounded and clearable A TTL bounds how stale an answer can get, and Clear cache regenerates everything at once when a prompt template, corpus or model changes upstream. ### Honest about the short-circuit A hit skips the firewall and the router because the stored answer already passed them. New policy applies to fresh requests, or after you clear the cache. ## Without the footguns. ### Deterministic by default Only reproducible requests are cached, those with temperature 0 or a fixed seed. Volatile requests pass through untouched, unless you opt in to caching them. ### Isolated and bounded Your cache is scoped to your organization, never shared across tenants, and bounded by a TTL you set from 1 hour to 30 days. Clear it whenever you want. ### Cheaper and faster A hit records a small cache charge and skips the inference charge. That skipped charge is your saving, and the answer comes back in milliseconds. ## Questions people ask about Cache. ### Will it ever return a wrong answer? Exact matching cannot: the request has to be identical. Semantic matching is optional, and you set the similarity threshold at which a paraphrase counts. ### What gets cached? Only reproducible requests by default, meaning temperature 0 or a fixed seed. A request that asked for variety passes through untouched unless you opt in. ### How stale can an answer get? As stale as the TTL you set, from 1 hour to 30 days. Clear cache regenerates everything at once when something upstream changes. ### Is our cache shared with anyone? No. It is scoped to your organization and never shared across tenants. ### Does a hit skip the safety checks? Yes, and deliberately: the stored answer already passed them. New policy applies to fresh requests, or after you clear the cache. ## Stop recomputing the same answers. Create a key, turn on Cache, and serve your repeat traffic from a stored answer in milliseconds.