星脉网络：面向GPU集群集合通信与集中式路由的协同优化

摘要：图形处理器（GPU）集群网络流量不断增加，运营难度明显加大，这给高性能大规模GPU集群网络系统的构建带来新的挑战与机遇。提出了一种能够实现超10万GPU集群互联的无损高性能网络方案——星脉网络。GPU集群网络需要联合优化端侧的集合通信库和网络路由控制器，以实现多路径的高效集合通信。为此，针对星脉网络研发了端侧集合通信库（TCCL）以实现最短的跨节点路径规划，同时还开发了全局优化路由器GOR以避免路径冲突导致的网络拥塞。在腾讯大模型GPU集群中，星脉网络方案和公开GPUs集群方案（NVIDIA NCCL）的对比结果表明：星脉网络可以实现25%的集合通信带宽提升，同时避免80%的由流量冲突造成的网络拥塞问题。

关键词：大规模GPU集群；集合通信；负载均衡

Abstract: The network traffic of the graphics processing unit (GPU) cluster is continuously increasing, and the operation difficulty has significantly increased, which brings new challenges and opportunities to the construction of high-performance large-scale GPU cluster network systems. A lossless high-performance network scheme—Astral Network is proposed, which can realize the interconnection of over 10 000 GPU cards. GPU-centric networks require joint optimization of the collective communication library at the host and centralized routing controller to achieve efficient collective communication over multiple paths. Therefore, Tencent developed a collective communication library (TCCL) for Astral Network to achieve the shortest path planning across nodes, and a global optimized router (GOR) to avoid network congestion caused by route conflict through traffic planning. In Tencent’s large-scale GPU-centric clusters, the comparison results between the Astral Network and the publicly available GPU-centric network (i.e., NVIDIA NCCL) show that: Astral Network achieves a 25% increase in collective communication bandwidth, while avoiding an 80% network congestion caused by traffic conflicts.

Keywords: large-scale GPU clusters; collective communication; load balancing