面向边缘场景的高效大模型分层部署机制

摘要：针对边缘设备资源受限导致大模型推理延迟高、吞吐量低的问题，提出了一种高效的分层部署优化机制，构建了“数据供给-空间保障-时间重叠”协同架构。具体而言，首先利用内存映射（mmap）实现权重按需加载以打破大模型加载的I/O瓶颈。其次通过静态内存预分配与指针偏移复用，彻底规避垃圾回收引发的系统抖动。最后提出基于权重矩阵粒度的流水线调度，拆解并行计算与加载任务，有效重叠显存访问延迟。实验表明，该机制在不增加显存占用的前提下显著提升了推理效率，推理吞吐量提升一倍以上，有效支撑了边缘侧大模型的实时推理需求。

关键词：边缘计算；大模型部署；推理优化；分层推理

Abstract: Addressing the issue of high inference latency and low throughput caused by resource constraints on edge devices, we propose an efficient hierarchical deployment optimization mechanism that establishes a collaborative architecture featuring “data supply—spatial assurance—time overlap”. Specifically, we first leverage memory mapping (mmap) to enable on-demand weight loading, thereby breaking through the I/O bottleneck associated with large model initialization. Next, by statically pre-allocating memory and reusing pointer offsets, we completely eliminate system jitter caused by garbage collection. Finally, we introduce a pipeline scheduling approach based on the granularity of weight matrices, which decomposes parallel computation and loading tasks, effectively overlapping the latency of graphics processing unit (GPU) memory access. Experimental results demonstrate that this mechanism significantly improves inference efficiency without increasing GPU memory usage, more than doubling the inference throughput and effectively supporting real-time inference requirements for large models at the edge.

Keywords: edge computing; large model deployment; inference optimization; hierarchical inference