Jingchao Zhang, PhD

Dense vs MoE: IB and GPU Communication Patterns

Introduction In my previous post, I showed that Mixtral 8x7B (MoE) requires 56× more effective interconnect bandwidt...

Monitoring IB Counters with Prometheus and Grafana

Introduction In my previous post, I showed that InfiniBand delivers 27–57× higher multi-node throughput than Etherne...

GPU Stress Tests Degraded InfiniBand Performance

The Problem

InfiniBand vs Ethernet for Multi-Node LLM Fine-Tuning

Azure Managed Lustre for GPU VMSS Clusters

The Problem: Every Node Needs the Same Data When you scale a GPU cluster beyond a single node, you immediately hit a...

Multi-Node Fine-Tuning: FSDP Sharding Strategy Matters

Introduction In a previous post, I ran Qwen2.5-72B inference on Azure H100 nodes and showed how NVLink’s 900 GB/s ba...