Jingchao Zhang, PhD

NCCL Ring vs Tree for Multi-Node LLM Fine-Tuning

TL;DR — We benchmarked NCCL Ring, Tree, and Default allreduce algorithms across 1–8 nodes (8–64 H100 GPUs) on Azure N...

Impact of GPU Thermal Throttling on LLM Training

Introduction A single thermally throttled GPU — one out of sixteen — can cut your distributed training throughput by...

Dense vs MoE: IB and GPU Communication Patterns

Introduction In my previous post, I showed that Mixtral 8x7B (MoE) requires 56× more effective interconnect bandwidt...

Monitoring IB Counters with Prometheus and Grafana

Introduction In my previous post, I showed that InfiniBand delivers 27–57× higher multi-node throughput than Etherne...

GPU Stress Tests Degraded InfiniBand Performance

The Problem

InfiniBand vs Ethernet for Multi-Node LLM Fine-Tuning