HPC/AI @Microsoft
We implement a ring allreduce algorithm from scratch in Python, run it on 16 NVIDIA H100 GPUs across 2 nodes with Inf...
Hands-on experiments on an NVIDIA H100 GPU reveal why KV cache — not model weights — dominates GPU memory during infe...
When scaling down an AKS node pool, which node gets deleted? The answer depends on how you trigger the scale-in and w...
TL;DR — We benchmarked NCCL Ring, Tree, and Default allreduce algorithms across 1–8 nodes (8–64 H100 GPUs) on Azure N...
Introduction A single thermally throttled GPU — one out of sixteen — can cut your distributed training throughput by...
Introduction In my previous post, I showed that Mixtral 8x7B (MoE) requires 56× more effective interconnect bandwidt...