KV-Cache as Attack Surface: Side Channels in Shared LLM Inference Infrastructure
About This Session
Most people think about LLM security at the prompt level. What goes in, what comes out. But there's an entire layer underneath, the KV-cache, and it has some properties that should make anyone running shared inference infrastructure genuinely uncomfortable.
Here's the problem: when a transformer model processes text, it doesn't recompute everything from scratch every time. It caches intermediate computations, the key-value pairs from the attention mechanism, so it can reuse them for efficiency. That's great for latency and cost. It's less great when multiple tenants share the same inference hardware, because those cached states don't always get cleaned up the way you'd expect.
This talk is about what happens in that gap.
We'll get into the mechanics of how KV-caches work, why they exist, and specifically where the isolation assumptions break down in multi-tenant deployments, including managed API services, on-premise inference clusters running vLLM or TensorRT-LLM, and enterprise platforms built on shared GPU pools.
We'll walk through a concrete demonstration on a locally hosted open-weight model, nothing proprietary, nothing that requires a vendor's cooperation to reproduce. The goal is to show this isn't theoretical. It's measurable, it's repeatable, and the conditions for it exist in a lot of production deployments right now.
The second half of the talk pivots to what you can actually do about it.
Here's the problem: when a transformer model processes text, it doesn't recompute everything from scratch every time. It caches intermediate computations, the key-value pairs from the attention mechanism, so it can reuse them for efficiency. That's great for latency and cost. It's less great when multiple tenants share the same inference hardware, because those cached states don't always get cleaned up the way you'd expect.
This talk is about what happens in that gap.
We'll get into the mechanics of how KV-caches work, why they exist, and specifically where the isolation assumptions break down in multi-tenant deployments, including managed API services, on-premise inference clusters running vLLM or TensorRT-LLM, and enterprise platforms built on shared GPU pools.
We'll walk through a concrete demonstration on a locally hosted open-weight model, nothing proprietary, nothing that requires a vendor's cooperation to reproduce. The goal is to show this isn't theoretical. It's measurable, it's repeatable, and the conditions for it exist in a lot of production deployments right now.
The second half of the talk pivots to what you can actually do about it.
Speaker
Saloni Garg
Senior Machine Learning Engineer - Adobe
Saloni Garg is a Senior Machine Learning Engineer at Adobe, where she specializes in AI and machine learning. She is also a recognized author in the field, with a forthcoming book on Retrieval-Augmented Generation (RAG) to be published by De Gruyter later this year. Saloni has made significant contributions to the academic community, evidenced by her multiple IEEE publications and a published patent.