Bringing the Large-Scale MoE Model GLM-5.1 to Practical Speeds on a Single NVIDIA H200 Node

“Optimizing Inference Performance for the Open-Weight LLM GLM-5.1” is a hands-on white paper for running large-scale MoE models comfortably on limited hardware.

The challenge this paper addresses

GLM-5.1, a large-scale MoE model with 754B total parameters, can be loaded onto a single node packed with eight NVIDIA H200 GPUs—but out of the box it won't deliver a practical response speed. The cause isn't the size of the model itself; it's the exhaustion of the memory left over for the KV cache. And simply adding more GPUs won't increase capacity if your parallelization strategy is wrong. This paper tackles that challenge, drawing the maximum response performance and serving capacity out of the available hardware resources.

What sets this paper apart
  • Optimization that digs into the architecture: Expanding KV cache capacity by introducing DP Attention, grounded in an understanding of GLM-5.1's characteristics—MLA, DSA, and sparse MoE.
  • Measures that raise serving capacity: KV-cache-aware request routing, and avoiding prefill recomputation through CPU cache offloading.
  • Quantitative evaluation based on real workloads: Benchmarks generated from actual coding-agent sessions, eliminating the "fast on the benchmark but slow in production" gap.
  • Clear results and implications for hardware selection: P90 TTFT reduced from 59 seconds to 3.8 seconds (roughly 1/15). Includes insights useful for selecting an inference platform for large-scale MoE models, drawn from comparative testing against a B200 environment.
Who this is for
  • Those who want to run the latest open-weight LLMs securely, on-premises or in their own environment.
  • Those who want to maximize the number of concurrent connections (serving capacity) on a limited GPU node.
  • Those considering their choice of inference platform, GPU, or quantization format for large-scale MoE models.
Download the White Paper