存储层级感知的大模型前缀键值缓存系统

摘要：随着智能体应用的发展，大语言模型服务中的前缀键值（KV）缓存容量需求不断增长，逐渐超出GPU显存和CPU内存的承载能力。利用容量大、成本低的SSD扩展前缀缓存具有较大潜力，但其延迟较高、随机读带宽较低，直接应用会显著增加首字延迟（TTFT），难以满足实际部署需求。针对这一问题，提出了存储层级感知的大模型前缀键值缓存系统PCache。PCache面向显存-内存-固态硬盘（SSD）分层存储架构，通过数据布局感知的重要KV选择、页面级连续KV加载、动态KV重排和变序计算等技术，减少SSD访问与随机输入/输出（I/O）开销。相关实验验证了PCache可以在保持推理精度稳定的同时降低TTFT。

关键词：大语言模型；前缀KV缓存；分层存储；首字延迟

Abstract: As the capacity demand of prefix key-value (KV) caches in large language model services for agent scenarios continues to grow, it has gradually exceeded what GPU memory and CPU memory can support. Although using high-capacity and low-cost SSDs to expand prefix caches is promising, SSDs have high access latency and low random-read bandwidth, and directly introducing them can significantly increase time to first token (TTFT), making it difficult to meet practical deployment requirements. To address this issue, this paper proposes PCache, a storage-hierarchy-aware prefix key-value (KV) cache system for large language models. On a GPU memory-CPU memory-solid state drive (SSD) hierarchy, PCache reduces SSD accesses and random input/output (I/O) overhead through data-layout-aware important KV selection, page-level continuous KV loading, dynamic KV reordering, and out-of-order computation. Experimental results show that PCache significantly reduces TTFT while maintaining stable inference accuracy, providing a technical reference for designing efficient prefix KV cache systems for long-prefix large language model serving.

Keywords: large language model; prefix KV cache; hierarchical storage; TTFT