长距高性能网络关键技术研究

摘要：针对AI大模型训练规模突破万卡、需跨园区长距组网的新需求，提出了一套端到端的联合优化解决方案：分级流控技术将任务完成时间（JCT）降低一个数量级；快速拥塞反馈机制将拥塞反馈延迟从毫秒压缩至数十微秒；逐流负载均衡协议消除端口负载不均问题；采用拓扑亲和的ZeRO 1策略替代ZeRO 3，大幅减少跨园区通信次数。实验网测试表明，通过整合上述关键技术，在千亿参数模型训练场景下可将跨园区训练的算力损失从10%~15%降低至5%以内，验证了长距网络支撑超大规模AI训练的可行性。

关键词：长距高性能组网；大模型训练；拥塞控制；负载均衡

Abstract: This paper proposes an end-to-end solution to address the new requirements of AI large-model training at the scale of over ten thousand GPUs and the need for long-distance, cross-campus networking. The hierarchical flow control reduces job completion time (JCT) by an order of magnitude. The fast congestion feedback mechanism compresses congestion feedback latency from milliseconds to tens of microseconds. Per-flow load balancing protocol eliminates port load imbalance. And the topology-aware ZeRO 1 strategy in place of ZeRO 3, can greatly reduces the number of cross-campus communications. Testbed experiments show that by integrating these key techniques, in the training of models with hundreds of billions of parameters, the compute efficiency loss of cross-campus training was reduced from 10%–15% to within 5%, validating the feasibility of long-distance networks in supporting ultra-large-scale AI training.

Keywords: long-distance high-performance networking; large-model training; congestion control; load balancing