Sixteen experts and an unexpected N

The failure

When SGLang runs a mixture-of-experts model with DeepEP, each GPU receives the tokens routed to its own experts, and the DeepGEMM runner then needs one small vector on the GPU: how many tokens each local expert received. Usually that vector is an ordinary pinned-memory copy from the host. In OffloaderV2’s CPU mode, SGLang does not use the copy engines for it and calls copy_to_gpu_no_ce instead. That function hands the values to a kernel as its argument, and the kernel writes them into GPU memory: the data travels with the launch.

The function knew three lengths, 32, 64, and 72, and stopped on anything else with RuntimeError: unexpected N. A valid configuration reaches another length. Split 256 experts across 16 expert-parallel ranks and each GPU holds 16, so the server failed with that error during prefill warmup (issue #32507). An earlier pull request, #23550, had fixed the same kind of fixed-length failure for 32.

Two ways to fix it

Adding 16 to the list would repair this configuration and leave the next one to fail. Replacing the list with one kernel large enough for any length repairs them all; another open pull request, #32515, does that. I kept both: an exact kernel where one exists, and a bounded fallback for every other length.

The fallback copies the values into a fixed array of 512 slots and passes the true length beside it, so one compiled kernel serves every length. The bound comes from the argument itself: 512 four-byte values make 2 KiB, which leaves room for the count and the output pointer under CUDA’s traditional 4 KiB limit on kernel parameters. Past the bound, the function now refuses the input and says why. The input checks moved into one function that every path shares.

if (N == 16) {
  copy_to_gpu_no_ce_impl<16>(input, output);
} else if (N == 72) {
  copy_to_gpu_no_ce_impl<72>(input, output);
} else if (N == 64) {
  copy_to_gpu_no_ce_impl<64>(input, output);
} else if (N == 32) {
  copy_to_gpu_no_ce_impl<32>(input, output);
} else {
  // Preserve the compact launch arguments for common
  // expert counts. Less common sizes pay for the bounded
  // fallback instead of using a copy engine.
  copy_to_gpu_no_ce_fallback_impl<kCopyFallbackMaxN>(
      input, output);
}

Checking it

The copy is exact: integers in, the same integers out. It never touches model arithmetic, so correctness reduces to equality. Each test fills a CPU tensor with random 32-bit integers, copies it to the GPU, and compares every element with PyTorch’s own copy. The lengths are chosen to reach each path and each edge:

• 16The new exact path, and the length that failed.
• 17The fallback, one past an exact length.
• 32, 64, 72The existing exact paths, which must not change.
• 512The fallback’s upper bound.
• 513Must be refused, with the bound in the error message.

All of them passed on an NVIDIA L40S with CUDA 12.8 and PyTorch 2.10. The change was also checked with py_compile, clang-format, and git diff --check.

What I measured

Keeping the exact kernels is a performance claim, so I measured it for the case that started the change: 16 values on an L40S.

The exact kernel’s latency was 7.6% lower than the generic kernel’s, and 47.3% lower than the pinned-memory copy’s.

What it does not show

These are microseconds for one small copy, repeated 20,000 times, on one GPU, at one length. They show that here the exact path is the fastest of the three. They do not show that serving gets faster: the pull request reports no prefill, decode, or end-to-end measurement.

The cause of the 150-nanosecond gap is also unmeasured. The generic launch carries 2 KiB of arguments where the exact one carries 64 bytes, which is the obvious suspect, but the benchmark does not separate it from anything else.

The coverage is narrow in the same way. Only the 16-value copy was timed, on one GPU model. The tests call the copy directly; the pull request does not include a run of the 256-expert, 16-rank configuration that first failed.

Whether 150 nanoseconds are worth a second code path is for the maintainers to decide. As of October 13, 2026, this pull request and #32515 are both open.

Sources: the pull request’s description, diff, and tests, and the SGLang repository as of October 13, 2026.