面向分布式推理的大模型键值缓存分发网络

摘要：大模型推理已成为智能算力体系的核心，传统云端集中式推理面临带宽压力大、推理延迟高、算力资源紧张等瓶颈。现有云边协同方案在通信开销与键值缓存（KV-Cache）复用效率方面仍存在明显不足。为此，提出一种基于内容分发网络（CDN）的大模型KV-Cache分发网络（KVCDN），将传统CDN升级为具备张量感知与状态管理能力的分布式缓存架构。该架构采用端侧语义规范化、边缘热点分级缓存与核心全局去重编排三层协同结构，并结合提示词（Prompt）标准化、分级存储预放置、全局去重融合与语义路由4项关键技术，从而显著提升KV-Cache复用率，降低冗余计算与推理延迟。

关键词：大模型推理；键值缓存（KV-Cache）；内容分发网络（CDN）；云边协同；缓存复用

Abstract：Large language model inference has become a core component of intelligent computing systems. Traditional cloud‑centric inference approaches face critical bottlenecks such as high bandwidth consumption, excessive latency, and intensive computational demands. Existing cloud‑edge collaborative schemes still suffer from significant communication overhead and low cache reuse efficiency of Key‑Value Caches (KV-Cache). To address these issues, a KV‑Cache distribution network based on content delivery network (CDN), termed KVCDN, is proposed. In this architecture, conventional CDN nodes are upgraded into a distributed caching system with tensor‑aware perception and state management capabilities. A three‑layer collaborative structure is adopted, consisting of semantic normalization at the end side, hotspot‑aware hierarchical caching at the edge, and global deduplication with orchestration at the core. Combined with four key enabling techniques, i.e., prompt standardization, tiered storage and proactive pre‑placement, global deduplication and fusion, and semantic‑aware routing, the KVCDN significantly improves KV‑Cache reuse ratio, while reducing redundant computations and inference latency.

Keywords: large model inference; Key‑Value Caches (KV-Cache); content delivery network (CDN); cloud-edge collaboration; cache reuse