Storage in the Serving Path: Giorgio Regni and Paul Speciale on Shared KV Cache and Disaggregated Inference.
Q1. Giorgio, you describe (*) the KV cache on ADI as “fast enough to sit in the serving path.” Traditionally, storage has been too slow for this role, which is why the KV cache has resided in GPU high-bandwidth memory (HBM). What has changed? Could you walk us through the data path from the decode GPU to ADI and explain which components have the greatest impact on latency – RDMA, NVMe, parallel streaming across the cluster, or something else? In your 14K-token test, warm time to first token was 166 milliseconds, just 83 milliseconds slower than HBM. Where does that additional latency come from, and how much of it can realistically be recovered?
Giorgio Regni: The first thing that has changed is the size of the KV cache, since it used to be small enough that keeping it in GPU memory was the easy and obvious choice. That stopped being true with the arrival of long-context models and agents. In our lab, one GLM-5.2 session with a 438,764-token context produced 189 GB of KV cache (about 430 KB per token). With the model loaded across all eight GPUs of the server, that one session does not fit anywhere but external storage. HBM is still the fastest place to hold a context, but it can no longer hold most of them.
The second change is the data path. On a read, the GPU reserves space in its own memory and tells Scality ADI which objects to put there, so the GPU never drives the transfer itself. Scality ADI has the TLC NVMe drive place the object in the storage server’s RAM over PCIe, and the network card sends it from there straight into GPU memory over RDMA. The data bypasses the CPU, the kernel and the software stack, which also frees the storage servers. On one 100 Gbit/s link (slow by today’s standards where 400/800Gbit is common), S3 over TCP moved 4.6 GB/s with 24 of the storage server’s 48 cores busy, while Scality ADI over RDMA moved 9.6 GB/s with one core busy, and 42.5 GB/s across four links with two. On the control path side, a standard S3 read carries the overhead of request signing, policy checks, metadata lookups, checksums and XML framing on every request. The Scality AI Connector replaces that with our own stripped-down control plane, with single-pass authentication, so the control path runs in microseconds while the object model stays the same. Each KV block is its own object, so a restore reads from many drives across many servers at once.
Of the components you listed, the network sets the limit in our lab, not the drives. In our testing, five storage servers (several generations old) never saturated their NVMe, and the four 100 Gbit/s RDMA links reached their ceiling first. The other large factor was fetch efficiency. The serving engine’s KV offloading connector was requesting 9 GB for a request that needed 1.7 GB. Once we fixed it, the amplification fell from 5.1x to 1.1x and warm time to first token dropped from around 360 ms to 166 ms.
Warm time to first token (TTFT) on Gemma-3 27B with a 14K-token context was 83 ms from HBM, 102 ms from the server’s own DRAM and 166 ms from Scality ADI over RDMA. Recomputing the same context took 2.3 to 3.0 seconds. Most of the gap to HBM is the time to move the context’s bytes across the fabric before the first token. A smaller part is per-block request overhead. More fabric bandwidth recovers the first part, and prefetching blocks while a request waits in the queue can hide some of it. Not all of it is recoverable, and we do not expect to match HBM. The useful comparison is the same order of magnitude as GPU memory, against seconds to recompute. Only the first token waits on storage. The whole fetch completes before token one, and every token after it runs from GPU memory, so inter-token latency did not change.
Q2. Disaggregated serving separates prefill from decode, allowing “any decode GPU to pick up any context.” For engineers who have not deployed this architecture, which workloads benefit most in practice? Could you share some concrete examples?
Consider long-running agentic coding sessions, multi-turn conversations that resume hours later, retrieval-augmented generation (RAG) with large shared prefixes, and systems that keep up to 1,000 concurrent sessions resumable, as you describe. Which of these use cases sees the greatest benefit, and where does disaggregated serving offer little or no advantage, for example, with short, single-turn prompts?
Giorgio Regni: In simplified terms, split serving helps to separate the two jobs because otherwise they get blended. In mixed serving, which is how most clusters run today, every GPU does both jobs. Each new prompt interrupts the answers already being written, every GPU keeps some headroom it never uses to hold its latency guarantee, and each GPU can only reuse the KV cache it built itself. Disaggregated serving puts prefill and decode on separate GPU pools, so each pool does one job full time and can be sized for it. The shared cache is what makes that split practical, since the KV cache sits in Scality ADI rather than inside one server, every GPU can reach it, so prefill writes the cache and any decode GPU can pick up any context. So the real enabler there is the sharing, not the speed. The speed matters for a different reason, which is that it makes reading a context back much cheaper than recomputing it.
Of your four examples, long-running agentic coding sessions will benefit the most. An agent re-sends the repository, the system prompt, the tool definitions and the history on every turn and adds a little. For example, when I type a one-word hello to the open-source agent on my own desktop, it sends 18,740 tokens: about 12,000 tokens of tool definitions, 3,000 of instructions and 4,000 of skills, memory and clock. Over four calls I typed 47 tokens, the agent sent 75,345, and the provider served 53,760 of them from its cache. Published measurements of agentic workloads put the share of input tokens reused between turns at 84.6% to 99.5%. The same research shows cache hit rates falling from about 98% to between 16% and 26% once too many concurrent agents crowd a cache that lives in GPU memory. That is exactly the case a shared tier fixes.
Conversations that resume hours later are next. In GPU memory, that context has long been evicted and gets recomputed. In Scality ADI it is still there, and capacity does not move first-token time much. In a separate run with one serving instance per GPU, we held 8 TB of KV cache, more than 80 times the memory of one GPU, and first-token time stayed between 355 and 373 ms at every size from 69 GB to 8 TB.
RAG with large shared prefixes benefits when the prefix is truly identical. Blocks are named by a hash of their tokens chained to the block before, so a shared system prompt or a fixed document set at the front of the prompt reuses well. Retrieved chunks that change order on every query break the chain at the first difference. The 1,000 resumable sessions figure is the multi-user version of the same point.
The gains are small for short, single-turn prompts with no reuse. There is nothing to read back, and recomputing a few hundred tokens is cheap, and we would say this to customers. The published gains from disaggregation itself are also largest with long prompts and strict latency targets. DistServe (OSDI 2024) reports up to 7.4x more requests per GPU within the same latency targets in its best case, while Microsoft’s Splitwise (ISCA 2024) reports 1.4x higher throughput at 20% lower cost. Those are third-party research results, not Scality measurements.
Q3. Paul, prefill and decode GPU pools can scale independently. How should operators determine the right balance between the two, given that prefill is primarily compute-bound while decode is constrained by memory bandwidth? Does this separation also allow operators to use different types of hardware for each pool?
The control plane also routes requests to the appropriate GPUs based on where the relevant context resides and schedules workloads against service-level agreement (SLA) targets. How does that routing decision work when the KV cache is shared across the cluster rather than stored locally on a particular server?
Paul Speciale: Prefill demand is driven by the tokens you actually have to compute. Decode demand is driven by how many sessions are simultaneously generating, and how long their answers run. A shared cache changes the prefill number a lot, since every returning request read back instead of recomputed gives the GPU back 2.1 to 2.8 seconds of prefill on a 14K-token context. On an agentic workload with high reuse, the prefill pool sizes for new and missed context, not for every token that comes in. Decode, on the other hand, sizes for concurrency and memory bandwidth. Published production systems show where that leads. Manus reports about 100 input tokens for every output token in its agent (Yichao Ji, July 18, 2025). DeepSeek’s published serving system ran 32 prefill GPUs on four servers against 144 decode GPUs on 18 servers (March 1, 2025). With high reuse, the decode pool ends up the larger of the two.
We therefore cannot hand customers a fixed ratio, because there really isn’t one. We ask them to bring their traffic and a repository, and we can help to size the GPUs, the fabric and the cache against it. The ratio also moves over time as the hit rate climbs, which is the practical value of scaling the two pools independently.
On different types of hardware: yes in principle. Prefill is compute-heavy and decode is memory-heavy, and once the two run on separate pools there is no technical reason they need identical GPUs. The constraint that matters more in practice is the network. In our lab, a single NIC generation mismatch between the GPU server and the storage servers degraded the rail facing the GPUs until we fixed it. Uniform NICs and a correctly configured lossless network class affected results more than the choice of flash.
On routing, the control plane authenticates and meters requests, routes them to the GPUs holding the relevant context and schedules work against SLA targets. With the cache shared, “holding the context” is no longer a routing hard requirement, but can still help. If a GPU still has a session’s blocks in its own memory, sending the request there is cheapest. If that GPU is busy, any other GPU can pull the context from Scality ADI and start, at a known cost. Routing then becomes a trade between locality and load against the SLA, rather than a hard binding of sessions to servers.
Q4. Giorgio, a shared, multi-petabyte cache introduces several important questions around data management and security. How are cache entries identified and reused? For example, can prefixes be matched across users or sessions, and under what conditions? What happens to the cached state when a model version or quantization configuration changes? How does eviction work across the TLC flash, hard disk drive (HDD), and optional tape tiers within a single namespace? Finally, KV-cache state is derived from prompts that may contain sensitive information. How do you enforce isolation between tenants, and how does that approach support the data sovereignty and control arguments for on-premises inference?
Giorgio Regni: The serving engine slices the KV cache into fixed-size blocks and names each one by a hash of its contents (i.e., the tokens), chained to the block before it. Identical prefixes produce identical names, which makes each block a natural object. The connector will save a block once it is computed and load it back before it is needed. On Scality ADI, each block is an object on TLC NVMe drives.
Since the names derive from content, two sessions with the same prefix therefore match by design. Whether they are allowed to is a policy decision: within one tenant, sharing a system prompt or a common document set across users is the whole point. Across tenants, sharing is never something we would enable; each tenant’s cache stays separate.
A KV cache is only valid for the exact weights and configuration that produced it, so after a model version or quantization change the old entries can’t be reused and simply age out.
On tiering, Scality ADI keeps one namespace across TLC flash, HDD and, optionally, tape, and a lifecycle policy moves objects between media inside the same bucket. For the KV cache specifically, the serving tier is TLC. Colder media make sense for model weights, enterprise data and aging (older) inference state. We would never suggest restoring a live context from tape. When an entry is evicted, the cost of the miss is a recompute, which is exactly what it would have incurred without the cache.
Indeed, the cache deserves the same care as the prompts since it is derived from them. The cloud providers already treat it as application state, for example OpenAI’s prompt caching guide says cached prompts may store encrypted key/value tensors in GPU-local storage, kept for up to 24 hours. For an agent, that can mean the organization’s source code and documents are cached and stored. By running on the customer’s own infrastructure, none of it leaves their perimeter, and it stays under their jurisdiction. The stack is open code, so a customer can inspect exactly how inference state is stored and moved rather than just taking our word for it. For a regulated organization, that ability to audit the path is as much a part of sovereignty as where the hardware sits.
Q5. Your results are striking: up to 14 times faster than recomputation at 14K tokens, up to 72 times faster at 439K tokens, and 97% of network line rate. For a skeptical engineer evaluating the architecture, where is the break-even point? At what context length does restoring a KV cache from ADI stop outperforming recomputation? What network fabric and GPU-to-storage connectivity did your tests assume, and what infrastructure would a team need to reproduce those results? Finally, how does the system behave on a cache miss or when the storage path is slower than expected? Does it fall back to recomputation, and if so, what are the implications for tail latency and SLA compliance?
Giorgio Regni: We published two points on the curve. The first is 14x faster KV cache retrieval than recomputation on a 14K-token context. The second is up to 72x faster on a 439K-token context, where recomputing took 529 seconds on GLM-5.2 and reading the context back took 7.3 seconds. This ratio grows with context because recompute cost grows faster than the bytes to move. Reading back has a roughly fixed per-request cost plus a transfer that scales with the number of tokens. Recompute scales at least linearly and becomes superlinear as attention dominates. The crossover point is at short contexts, where recompute already takes well under the round trip to storage. We have not published that point; it depends on the model, the hardware and the fabric. Below it, there is little reason to read back, and the system should do the recompute.
The lab was one server with eight GPUs, five storage servers (again, the lab had servers several generations old) and four 100 Gbit/s RDMA links on a lossless RoCE class. The network was the limit, not the drives, and we reached 97% of network line rate for data transfer between GPUs and storage. The 166 ms figure used Gemma-3 27B, about 14,000 tokens of context and a KV block size of 256. To reproduce it, a team needs an RDMA fabric with a lossless class configured end to end, the same NIC generation on both sides, TLC NVMe storage servers and the open-code stack, which is public, with every fix committed upstream. We publish the model, context length, block size, link count and speed and hardware generation so anyone can reproduce or dispute the results. These are our own lab numbers, not independently validated ones, and we would welcome third-party testing.
On a miss, the request just falls back to a normal prefill. The cost is what it would have been without the cache, plus the lookup. When storage is slower than expected, first-token time moves toward the recompute figure, and the control plane schedules against SLA targets rather than assuming every restore is fast. We have not yet published tail-latency figures under a burst of misses or a degraded storage path.
Q6. Disaggregated serving and shared KV caches are evolving rapidly, alongside research systems such as DistServe and Mooncake. Looking across the open-source inference ecosystem, where do you expect standard interfaces for KV-cache storage and retrieval to emerge, and how is AI Inference Factory positioning itself to remain compatible with them? Looking three years ahead, which part of this architecture do you expect to change the most? And what experiment or benchmark would you most like to see the industry tackle next?
Giorgio Regni: Two interfaces are already beginning to settle down. One is at the serving engine, where open-source engines expose a KV connector interface that decides which blocks to save and load. The other is the object model underneath. Content-hashed, fixed-size blocks map naturally onto objects, and the pattern has moved quickly from research into production. Moonshot AI’s Mooncake paper described split pools with one shared KV cache in June 2024, and DeepSeek published its split serving system in March 2025. AWS, Google Cloud and Azure published split-serving recipes for their customers between July and October 2025, and Meta described split serving for its own large-scale traffic in September 2025. The layer still in flux is data movement between GPU memory and storage over RDMA, where the open-source libraries are evolving quickly. Our approach to compatibility is to have no fork. Every fix our testing needed above the storage layer went upstream, and our connector plugs into an existing open-source transfer library on its existing wire format. When the interfaces move, we move with them in public, and any object storage solution on the same path will benefit, competitors included. The architecture is also a familiar one for us. Keeping application state in distributed storage so any server can serve any user is what Scality RING has done since 2010 for email, backup and cloud storage, and we have a pending patent application on application session state on an object storage service.
In three years, I expect the biggest change in how the cache is managed rather than in how fast it is read. Today it is treated as an offload. It will become a managed tier with its own policies for sharing, retention, eviction and tenancy, the way any other enterprise data is managed.
Paul Speciale: I would add that once the cache is a tier, it is also data that an enterprise has to govern just as with all its mission-critical data. It is derived from prompts, code and documents, so retention, jurisdiction and audit apply to it just as they do to the source material. Organizations will place it deliberately, the same way they are starting to place inference itself, by jurisdiction and by economics.
On benchmarks, the industry has no standard benchmark for KV cache offload to external storage today, and MLPerf does not cover it. We would like to see a benchmark built on realistic multi-turn and agentic traces, with cache hit rate as the key metric (operators already treat it that way and also have reduced pricing for cached hits). Anthropic’s Claude Code team has said it runs alerts on prompt cache hit rate and declares production incidents when it falls too low (source: Thariq Shihipar, quoted by Simon Willison, February 20, 2026). It should report time to first token against context length, cache hit rate under rising concurrency, and tail latency on misses and on a degraded storage path. It should also publish the full conditions so results can be compared honestly. Vendors publishing speedup ratios from different models, GPUs and context lengths cannot be compared, and a common method would certainly help buyers more than any single vendor’s number.
Resources
(*) Scality launches AI Inference Factory to bring enterprise AI on-premises
……………………………………………………….
Giorgio Regni, Chief Technology Officer, Scality
As Co-founder and CTO at Scality, Giorgio lead technology strategy, architecture, and innovation for distributed storage platforms designed for AI data pipelines, cyber-resilient protection, and cloud-scale applications. His work sits at the intersection of object storage, distributed systems, performance engineering, and the leadership required to turn hard infrastructure problems into products customers trust.
Paul Speciale, Chief Marketing Officer, Scality
Over 20 years of experience in Technology Marketing & Product Management. Key member of team at four high-profile startup companies and two fortune 500 companies.
Sponsored by Scality