A Field Guide to Training–Inference Corrections

Your rollout engine and your trainer do not define the same policy — even with bit-identical weights. The field’s default response is a blanket importance-sampling correction, and the usual defense is “our runs are stable.” Both are mistakes outside a shrinking benign regime: eliminate the mismatch you can, then correct what remains. The eliminations come in three families (specified divergences, discrete selections, arithmetic path differences), and their costs form a triangle: bandwidth, approximation, engineering. This post is the short map; the companion post decides what the residual deserves.

1. Forking Paths: Correction vs Elimination

In RL we sample rollouts with an inference engine and learn by backpropagating through log-probs recomputed by a training engine. Those are different software, so they compute different log-probs. Most practitioners will grant that much without argument; the disagreement is over what it costs and what to do about it.

Weight differences and operator differences It is worth separating from what it resembles, because the two are routinely conflated. The familiar off-policy sources are weight differences: the rollout came from a policy some number of updates behind the one being trained. Those carry a version number and have standard corrections: multi-epoch drift, async staleness and its rate-limiting and decoupled-PPO treatments, catalogued in an earlier post that remains the reference for them [1]. The mismatch this post is about is what survives with the version number pinned. Load one checkpoint into both engines and their paths still fork in three places: different kernels round differently, discrete choices flip at ties, and under truncated sampling the rollout engine is not even drawing from the distribution the trainer scores. Any importance ratio built from those log-probs is supposed to carry policy movement and nothing else. At identical weights it should pin to exactly 1. On a mismatched stack it also carries everything the two engines disagree about.

Attacking the difference So the paths fork a second time, and this fork is yours to choose: correct the contaminated ratio, or remove the contamination. The field defaults to the first, and not unreasonably: it is nearly free to adopt. A TIS-style cap, a clip, a rejection band is a few lines in the loss function. No extra bytes on the wire, no kernel work, no contract to negotiate between two engines that different teams often own. Every elimination in this post, by contrast, asks something of the stack.

But a ratio treatment processes the symptom. It cannot repair a division that was never valid: a token outside the sampled nucleus has no ratio, only an error. It cannot reattach an expert that a rounding difference flipped. And every treatment has a price. Caps bias the estimate; clips and bands delete gradient signal. Removal is the only move on the board that pays neither.

The usual defense of the blanket correction is stability: “we run TIS and our training does not collapse.” Stability is a floor, not a ceiling. A blanket correction leaves no alarm for what it costs. Deleted signal does not spike any dial you monitor. It quietly lowers what the run converges to, and the loss shows up only if you run the counterfactual. TogetherAI’s XoRL ran exactly that experiment on a Wordle benchmark: a clamp-corrected run converged at 72.5% held-out, where the zero-mismatch run reached 77.4% [2].

And a single instrument cannot be right for the whole problem, because the mismatch is not one phenomenon. Part of it is a known transformation you chose. Part is a discrete flip with no valid first-order treatment. Part is genuine noise.

Which brings us to the map.

2. Three Families of Fixes

Specified divergences: remove exactly, never manage.

• Truncated sampling: reconstruct it. Under top-p top-k sampling configurations the sampler draws from a renormalized nucleus. For tokens the truncation excluded, the importance ratio is not large, it is undefined. A support mismatch sits outside any weighting frame entirely. The fix is exact reconstruction: capture the sampler’s keep-mask and renormalize the trainer’s distribution over the same nucleus. [3]

• Precision flows: honor them by construction. If the rollout engine quantizes to FP8, make the training forward see the same quantized policy rather than correcting for the gap afterward. Jet-RL does this with a unified FP8 flow [4], DeepSeek-V4 with FP4 and simulated-quantization training [5], and the same contract now carries onto Blackwell-native NVFP4. Quantize a tensor differently on the two sides and “the policies used for sampling and learning are no longer the same low-precision model”; match them and the BF16 reward curve holds while rollout runs faster [6]. The exact fix here therefore has negative cost: a unified flow runs the cheap format on both sides and gains throughput over the full-precision baseline while holding its quality. Consistency is the rare fix that pays for itself.

Discrete selections: replay or pin. MoE expert routing and sparse-attention index selection are hard argmax-like choices made under the sampler’s arithmetic. When the trainer re-makes them under its own, a rounding-level logit difference becomes a different expert or a different attended context. The deviation is bimodal and large rather than continuous and small, which is where a first-order ratio treatment is least valid. Two verbs apply:

• Replay the choices: record what the sampler picked and force it at recompute time. Exact. Routing replay is the canonical example [7], [8], with DeepSeek’s Keep Routing in the same class [3].
• Pin the selector: make it deterministic and freeze it so both engines provably agree. GLM-5 froze its sparse-attention indexer and switched to a deterministic top-k after the nondeterministic operator degraded RL within steps [9].

Arithmetic path differences: unify the arithmetic. What remains once supports match and selections agree is floating-point path difference: reduction order, kernel variants, batch-shape-dependent accumulation. Two typical cases:

• Batch-dependent GEMMs. A GEMM kernel picks its tiling and split strategy from the shape it is handed. The same sequence scored in a batch of 8 and in a batch of 512 accumulates in a different order, and lands on different bits. Nothing about the token changed, only its neighbours.
• The LM head’s vocabulary reduction. Every log-prob ends in a reduction across the whole vocabulary: a logit minus a log-sum-exp over a vocabulary that now runs to six figures, where the summation order belongs to the kernel rather than to the model.

Neither is random; both are reproducible the moment the two engines agree on how to add. This is the family that was most opaque and deemed most costly to remove, and it is increasingly neither. Horace He’s post named one of the root causes, batch invariance in the kernels rather than GPU nondeterminism, and shipped reference kernels for it [10]. The vLLM and SGLang communities have active efforts in the same direction. SkyRL’s IsoExec pins an execution contract across the trainer and vLLM, cutting the mean rollout-versus-training logprob gap from 1.6 × 10⁻² to 6.7 × 10⁻⁷ at ~25% end-to-end overhead [11]¹. A TorchTitan and vLLM effort reaches the same place from another direction, holding one model definition across both engines and running the recurrent kernel for both prefill and decode [12]. XoRL catalogs the strategy space: reuse the inference kernel inside the trainer, write one kernel for both engines, pin reductions batch-invariant, or run dual kernels with matched arithmetic. It reports bitwise-identical logprobs between its trainer and its SGLang fork at roughly 20% end-to-end step overhead [2]. On same-architecture hardware, sharing batch-invariant kernels between the rollout engine and the trainer’s scoring path achieves bitwise logprob parity, in BF16. The discipline is identical reduction order, not higher precision. Three things temper it:

• Price the training path separately. Batch-invariant kernels are rarely the fastest available, and the trainer pays the difference on every step.

• FP16 scoring is not automatically a cheap partial step. 8× the mantissa of BF16 at the same width looks like an easy win, and one line of work argues that simply reverting to FP16 eliminates the mismatch outright [13]. Reception has been mixed, for the reason this whole family turns on: precision is not the lever, the execution path is. Changing dtype does not align two reduction orders by itself, and a particular FP16 kernel can carry a larger gap than the BF16 one it replaced. Newer operators also often ship no FP16 kernel at all, and where one exists it is not guaranteed to be fast.
• One boundary is close to absolute; the other is expensive. Cross-architecture bitwise parity is impossible, so a heterogeneous fleet turns this family back into a floor whatever you do to the kernels. The prefill/decode seam is the expensive one. Training is a pure prefill workload while generation alternates prefill and decode, and for linear-attention layers that means a chunkwise-parallel form on one side against a recurrent one on the other. The two are mathematically identical, and IsoExec measured a maximum per-element difference of 0.25 between them [11]. Closing it means pinning the shape of the reductions on both sides. Sparse attention and context-parallel invariance complicate things even further. Where these boundaries bind, this family degrades back into a floor you correct rather than remove. It is the one setting where correction is genuinely the last resort rather than the first.

3. The Triangle: Bandwidth, Approximation, Engineering

Nothing above is free; the currencies differ. Replay is exact and pays in bandwidth. The sampler’s intermediate choices must ship to the trainer, and they are heavy: routing decisions dominate the generation payload for MoE replay, and top-p nucleus arrays ran to tens of times the rest of our training data [8], [14]. Pinning is bandwidth-free and pays in approximation. GLM-5 rejected index replay because its selector picks 2048 entries per token per layer where MoE routing picks ~8. It froze the selector instead, accepting that it can no longer adapt during RL [9]. Kernel unification is exact and bandwidth-free and pays in engineering. Shared batch-invariant kernels across two engines are real work, and the investment is capped by fleet homogeneity.

The triangle also has a degenerate fourth corner, easy to forget because it costs nothing to run and everything to admit: delete the component. It is exact and free, available exactly when the component is optional, and priced in whatever the component was buying you. That is where our top-p correction ended: the exact fix worked, and removing its reason to exist worked better.

4. Where to Go Deeper, and What Remains

Each stop on the map above has a fuller treatment elsewhere:

• Weight-difference mismatch (multi-epoch drift, async staleness, rate-limiting, decoupled PPO): the original corrections post [1].
• Routing replay, and what shipping the sampler’s choices actually costs: [8].
• Top-p mask replay: the build, the verification, and why we reverted it [14].
• The residual, and the criterion for treating it: the companion post [15].

After the toolbox, the ratio is still not clean. What survives is a residual with structure: heavy-tailed, concentrated on exactly the tokens where the learning signal lives, and reshaped by the very corrections you installed. There is a benign regime where a blanket correction over this residual is genuinely fine — easy tasks, short runs, short contexts — and it is exactly the regime frontier RL is scaling out of. So the last instruction of this post is the first instruction of the next one: measure the residual before you choose its treatment. The companion post gives it a decomposition, a signal-to-noise criterion, and a procedure to run on your own stack [15]. This post shrinks the mismatch; the companion decides what the survivor deserves.

References

[1] Off-Policy Corrections in LLM RL Training. March 2026.

[2] Panda, A. “0 train-infer mismatch for Open-weight MoE RL in Open-source code.” TogetherAI, August 2026. XoRL.

[3] DeepSeek-AI. “DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models.” December 2025, §3.1.

[4] Xi, H., Ruan, C., Liao, P., et al. “Jet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flow.” NVIDIA / MIT / UC Berkeley, January 2026.

[5] DeepSeek-AI. “DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence.” April 2026, §3.4, §5.2.1.

[6] Li, Z., and LMSYS Org. “Blackwell-Native 8-bit and 4-bit RL.” July 2026.

[7] Zheng, C., Dang, K., Yu, B., et al. “Stabilizing Reinforcement Learning with LLMs: Formulation and Practices.” Qwen Team, December 2025.

[8] The Infrastructure Cost of MoE Routing Replay. April 2026.

[9] GLM-5 Team, Zhipu AI & Tsinghua. “GLM-5: from Vibe Coding to Agentic Engineering.” February 2026.

[10] He, H. “Defeating Nondeterminism in LLM Inference.” Thinking Machines Lab, September 2025.

[11] Jiang, A., and the SkyRL Team. “IsoExec: Unified Execution to Eliminate Trainer-Inference Mismatch in SkyRL.” August 2026.

[12] Wang, Y., and the TorchTitan team. “Defending Against the Training–Inference Numeric Mismatch in RL (Especially Linear Attention) — and Whether It Helps Async RL.” August 2026.

[13] Qi, P., Liu, Z., Zhou, X., et al. “Defeating the Training-Inference Mismatch via FP16.” October 2025.

[14] Top-p Mask Replay. (Coming soon)

[15] Signal or Noise? The SNR Criterion. (Coming soon)