Greg Banks

KV-Cache Exploits and The Next AI Frontier: Academia is Building Defenses, While Industry Chases Performance and Cost 2025 is the year LLM serving grew up: vLLM, SGLang, and TensorRT-LLM turned “run the model” into a full-stack system problem. Disaggregated prefill vs. decode, FLOPs vs. bandwidth, and KV cache + scheduling policies became the centerpiece for user KPIs like throughput and tokens per $. As someone who’s spent a decade exploring the interplay of HW/SW, this sounds familiar: performance always came first, we love caching! But what about the hidden KPIs: security and privacy? The truth: a KV cache isn’t just an optimization buffer. It’s the working memory of prompts, and the closest thing to the “real intent" of users in the system.  When we introduced cross-tenant prefix sharing, cache-aware schedulers, or offloading, we created observable signals that leak what other users (or the system prompt) said. PROMPTPEEK[1] demonstrates prompt reconstruction by exploiting cache-aware scheduling and request completion times. “The Early Bird Catches the Leak” [2] shows timing side channels (e.g., TTFT diffs on cache hit vs. miss) that are used to infer system prompts and even other users’ prompts. InputSnatch [3] extends the idea to “input theft” by combining candidate construction with robust timing analysis under noise. It’s also showing in the wild: “Auditing Prompt Caching in Language Model APIs” [4] proposes audits that detect prompt caching and cache-sharing scope and evidence of cross-user/global sharing in *real* providers with latency silently becoming a privacy oracle. On the defense side, mitigations are still just papers and prototypes: per-tenant cache isolation (effective, expensive), selective sharing like SafeKV (share non-sensitive KV, isolate spans), timing obfuscation/constant-time strategies, and KV-cache protection schemes like KV-Cloak (for stronger “KV is exposed” threat models).[5-7] Call to action: if you start an LLM infrastructure company, treat KV cache as sensitive state, not a free performance lever. Productize what academia demonstrates: auditable caching policies, isolation tiers, privacy-aware cache managers, and security-first scheduling. The next generation of LLM winners won’t just ship the best tokens/$, but the best tokens/$ *with privacy guarantees*. [1] Wu et al. “I Know What You Asked: Prompt Leakage via KV-Cache Sharing in Multi-Tenant LLM Serving” 2 Song et al. “The Early Bird Catches the Leak: Unveiling Timing Side Channels in LLM Serving Systems” 3. Zheng et al. “InputSnatch: Stealing Input in LLM Services via Timing Side-Channel Attacks” 4. Gu et al. “Auditing Prompt Caching in Language Model APIs” 5. Chu et al. “Selective KV-Cache Sharing to Mitigate Timing Side-Channels in LLM Inference” 6. Luo et al. “Shadow in the Cache: Unveiling and Mitigating Privacy Risks of KV-cache in LLM Inference” 7. Yuan et al. “SCX: Stateless KV-Cache Encoding for Cloud-Scale Confidential Transformer Serving”

15

CPU overclocking in low-latency systems is usually more of a vanity metric than real edge. An extra 100–200 MHz looks great in the BIOS and in benchmarks. Production does not care about the peak clock on your screenshot. Production cares whether the CPU can hold a stable frequency under real burst load without thermal throttling, power limits, or extra jitter. If an overclock improves a synthetic benchmark but makes your p99/p99.9 worse at the market open, that is not optimization. That is self-inflicted noise. In low-latency systems, the best overclock is not the highest one. It is the one that does not wreck your production latency profile. That said, if you really do need to push clocks, we know a few tricks, feel free to Direct Message. #HFT #MFT #Linux #LowLatency #SRE #TradingInfrastructure #HighPerformance

41

8 Comments

A cluster at a tier-1 bank in ran at peak CPU for more than three months, and it went unnoticed. No alert. Nobody picked it up ... yes bad, yes we have all been there. Then one server went down, then the next, and it caused absolute havoc. The team and visualisations knew the box was busy. It never told why. That is the gap continuous profiling closes. OpenTelemetry now ships it in alpha: every process on the host, sampled all the time, including code you never wrote. You are already paying for every one of those cycles. Link in comments. #Observability #SRE #OpenTelemetry #ContinuousProfiling

For those pushing the boundaries of performance, a critical bottleneck often lies not within CPU cycles, but in memory bandwidth utilization. While cache hits are vital, many advanced applications remain memory-bound due to inefficient data access patterns. Random memory access severely hampers modern CPU architectures, which are designed for speculative execution and prefetching based on predictable, sequential data streams. When algorithms necessitate constant, unpredictable jumps in memory, the memory controller and CPU prefetchers are rendered ineffective, leading to significant stalls and underutilized bandwidth. Strategically re-designing data structures and algorithms to promote *contiguous memory access* can fundamentally transform performance. This architectural shift enables optimal memory streaming, allowing prefetchers to operate at peak efficiency and often yielding 2x-5x throughput improvements in computationally intensive, data-bound scenarios. This is a non-trivial but highly impactful optimization for true high-performance computing. #PerformanceEngineering #HPC #SystemDesign #Optimization #MemoryManagement #AdvancedCoding

1

GIGANETS externalizes KV cache to remote memory via GPUDirect RDMA 🔧 Stateless LLM serving just got a lot more practical. The system decouples compute from state storage by letting GPUs directly access remote memory pools for key-value caches during inference. No CPU involvement. No local GPU memory bloat. Traditional LLM serving architectures tightly couple KV cache with GPU compute, which creates scaling headaches when you're managing thousands of concurrent sessions. GIGANETS treats the GPU as a stateless inference engine: • KV cache lives in disaggregated remote memory • GPUs read/write via RDMA without CPU mediation • Each inference request pulls only what it needs • State persistence happens outside the GPU cluster This matters because KV cache is the memory bottleneck in long-context serving. Most architectures waste GPU HBM storing conversation history that could live elsewhere. The RDMA path is critical here—traditional network stacks add 10-50μs of latency per cache access. GPUDirect RDMA keeps it under 2μs. We're seeing a clear trend: disaggregation for LLM infrastructure. NVIDIA's already enabled the primitives. The question is whether network bandwidth can keep up with generation throughput at scale. Are you running into KV cache memory pressure before you hit compute limits? #LLMInfrastructure #GPUCompute #AIEngineering #MachineLearning

1

AegisBPF is an eBPF agent that actually blocks unwanted file and network access at the Linux kernel level, instead of only alerting you after something already happened. More: https://ku.bz/wd7SCHC3l

5

Glory be! Abingdon-based ‘fabless’ ASIC (Application Specific Integrated Circuit) chip maker EnSilica has turned a half-time pre-tax profit. OK, it was only £120k on (greatly improved) revenues of £12.7m, and the bottom line was still underwater. But this was enough for CEO Ian Lankshear to foresee “a clear path to profitability”. Cash burn was also down, which prompted CFO Kristoff Rademan to forecast positive cash flow “on operational monthly basis” by the end of this calendar year. Rademan also advised that full year revenues were pretty much in the bag (well, what he actually said was “95% contracted”) and reckoned they’d win orders worth at least another $100m over the next few months. So why didn’t investors jump for joy? EnSilica’s shares had already risen from 45p to over 51p a few days ahead of yesterday’s results announcement (from crystal-ball gazing, I assume) but have since wandered back down to 49p as I write, i.e. 1p below EnSilica’s IPO price back in May 2022. I guess you’ll have to ask investors about that yourselves – I’m not in that game anymore. But I do see a potential challenge to EnSilica’s growth plans based on something Rademan mentioned on the analyst call. He explained that Consulting revenues had declined during the first half of the year because they had to pull consultants off the job to work on NRE (non-recurring revenue) contracts. This flags up a skills availability issue, which I am hoping EnSilica’s recruitment plans will address. Why do I tell you all this? Well, I’ve said it before and I’ll say it again. EnSilica is important! It’s one of the very few UK semiconductor companies on the London Stock Exchange and it needs to succeed if it is to become a role model for UK semiconductor startups to emulate and aspire to. The market opportunity is clearly there for EnSilica to capture its fair share. But success will, as ever, come down to execution, execution, execution.

66

1 Comment