AIFIRST AI at Home Part 2: Multi-GPU Drifting
The brief
A hands-on technical writeup on running LLM inference across multiple AMD Radeon Pro V620 GPUs in a home server.
Key points
- The author found that layer parallelism hurt performance, dropping from 19-20 tokens per second on a single card to 12-13 across four cards, while speculative decoding roughly doubled speeds to about 40 tokens per second on a single card.
- PCIe 3.0 with 8 lanes created bottlenecks for inter-GPU communication, with peer-to-peer transfers and BIOS tuning improving tensor parallelism across two cards.
- Models tested include Gemma4-31B and Deepseek V4 Flash.
Sources
- HNjdagostino.github.io