BidKV：显存受限推理的KV缓存回收机制

摘要： 在线大模型推理中，键值（KV）缓存是制约并发服务的关键显存资源。在显存压力下，顺序回收或容量贪心回收易抢占高重算代价请求，导致释放的显存空间被重复预填充占用，从而放大首Token延迟（TTFT）。为此，提出BidKV，将KV回收建模为存算权衡问题，依据可释放容量与扰动代价选择单位代价收益最高的请求进行回收。BidKV不改变推理结果，并复用虚拟大语言模型（vLLM）与SGLang的原生抢占路径。ShareGPT实验结果表明，BidKV将vLLM的P95的TTFT由6 769 ms降至631 ms，300 ms 服务等级目标（SLO）达成率由68.4%提升至83.2%；在SGLang上，P95的TTFT较原生最近最少使用（LRU）降低94.6%，SLO达成率提升38.8个百分点。在双卡昇腾910B2（TP=2）实验中，BidKV在3.8请求/秒负载下实现了631.8 ms的TTFT P95与72.0%的SLO达成率。上述结果验证了BidKV的有效性、跨框架可移植性与多卡适用性，表明显存受限推理应从容量驱动转向存算协同调度。

关键词：大模型推理；大语言模型服务；KV缓存；请求调度；受害请求选择

Abstract: In online large language model (LLM) inference, the key-value (KV) cache is a critical memory resource that constrains concurrent serving capacity. Under memory pressure, sequential or capacity-greedy eviction policies tend to preempt requests with high recomputation costs, causing the freed memory to be repeatedly occupied by redundant prefill operations and consequently amplifying time-to-first-token (TTFT) latency. To address this issue, BidKV is proposed, which formulates KV cache eviction as a memory-computation trade-off problem. BidKV selects the request with the highest unit-cost benefit for eviction based on its releasable capacity and the corresponding perturbation cost. BidKV does not alter the inference results and reuses the native preemption paths of virtual large language Model (vLLM) and SGLang. Experimental results on the ShareGPT dataset demonstrate that BidKV reduces the P95 TTFT latency of vLLM from 6 769 ms to 631 ms and improves the 300-ms service-level objective (SLO) attainment rate from 68.4% to 83.2%. On SGLang, BidKV reduces the P95 TTFT latency by 94.6% compared to the native least-recently-used (LRU) policy and improves the SLO attainment rate by 38.8 percentage points. In a two-card Ascend 910B2 (TP=2) setup, BidKV achieves a P95 TTFT of 631.8 ms and an SLO attainment rate of 72.0% under a workload of 3.8 requests per second. These results validate the effectiveness, cross-framework portability, and multi-GPU applicability of BidKV, demonstrating that memory-constrained inference should shift from capacity-driven to memory-computation co-scheduled strategies.

Keywords: LLM inference; LLM serving; KV cache; request scheduling; victim selection