新架构大模型推理的存算特征演变与近存计算适配技术研究

摘要：多头潜在注意力（MLA）与混合专家（MoE）正在重塑大模型推理的存算特征：MLA通过低秩键值（KV）联合压缩，使权重吸收模式下的注意力算术强度由传统多头注意力（MHA）解码的约1 Op/Byte提升至数百Op/Byte；MoE则因实际批量下的专家权重访存瓶颈，成为近存计算的新核心场景。系统梳理解码阶段算术强度的解析模型以及“MHA→分组查询注意力（GQA）→MLA”的存算特征演变规律，对比分析MLA双执行模式的算术强度差异与现有存内处理（PIM）设备的算力密度约束，按临界批量模型对4类主流AI加速器进行解析估算与横向研判，并结合FP8/INT4等低精度计算特征，提出面向MLA+MoE组合架构的分层近存计算适配方向并给出框架性建议。

关键词：多头潜在注意力；混合专家模型；存算一体；近存计算；算术强度

Abstract: Multi-head latent attention (MLA) and mixture of experts (MoE) are reshaping the memory-compute characteristics of large language model (LLM) inference. Through low-rank joint compression of key-value (KV) representations, MLA raises the arithmetic intensity of attention under the weight-absorption mode from approximately 1 Op/Byte for conventional multi-head attention (MHA) decoding to several hundred Op/Byte, whereas MoE introduces an expert-weight memory-access bottleneck at practical batch sizes and becomes a new target for near-memory computing (NMC). This paper systematically reviews analytical models of arithmetic intensity in the decoding stage and investigates the evolution of memory–compute characteristics along the path of MHA → grouped-query attention (GQA) → MLA. It further compares the arithmetic-intensity characteristics of the two MLA execution modes and analyzes them in conjunction with the compute-density constraints of existing processing-in-memory (PIM) devices. Based on a critical-batch-size model, analytical estimations and cross-platform evaluations are conducted on four representative AI accelerators. Building upon these observations, the paper incorporates the characteristics of FP8/INT4 low-precision computation, proposes a hierarchical near-memory computing adaptation framework for combined MLA+MoE architectures and provides corresponding architectural recommendations.

Keywords: multi-head latent attention; mixture of experts; processing-in-memory; near-memory computing; arithmetic intensity