Signal or Noise? An SNR Criterion for Trusting Your Importance Ratio

Your importance ratio is a measurement, and like any measurement it contains signal (genuine policy movement) and noise from how two engines evaluate the same policy. The noise is not a blur: it is heavy-tailed, architecturally amplified, and aimed at exactly the tokens where the learning signal lives. This post shows its structure, gives a procedure to measure your own signal-to-noise ratio, and derives what the measurement licenses: which corrections to trust, which to bound, and which to never let delete a token.

1. The Two Sides of the Importance Sampling Ratio

Truncated importance sampling and its variants have become default-on machinery across RL frameworks. It is not guaranteed to save every run, though. Flash-RL claims that it enables FP8 rollouts [1]; Jet-RL calls the same fix “very fragile,” collapsing on long rollouts and hard tasks [2]. Where along the axis of context length and task difficulty does TIS start to break?

This post’s claim is that the unmeasured quantity has a name, a decomposition, and a procedure: the signal-to-noise ratio of the importance ratio itself.

Write the observed per-token log-ratio as the sum of two quantities with entirely different characters:

α is signal. It is the log-ratio between the training policy and the policy that actually sampled the token, and it goes nonzero whenever gradient updates separate the two. Two things separate them: asynchronous staleness and multiple optimizer steps per rollout batch. The second is easy to overlook. Modern frameworks split the batch into mini-batches consumed once each, so the k-th mini-batch trains under weights that have moved k−1 updates since sampling. These are the weight-difference sources catalogued in Off-Policy Corrections in LLM RL Training [3]. α is the thing importance sampling and trust-region machinery were designed for: if the policy has genuinely moved, the token’s gradient should be reweighted or bounded accordingly.

β is noise. It is what remains with bit-identical weights. The two log-probabilities are computed by different engines: different kernels and reduction orders, different precision paths, different discrete choices (expert routing, cached-KV versus recomputed context). These are the operator-difference families of A Field Guide to Training–Inference Corrections [4]. β is not a sampling-distribution error. The sampler really did sample from its own distribution; β is measurement error on the ratio. The instrument reads wrong, not the world.

Every ratio-conditioned intervention (importance weighting, TIS truncation, PPO-style clipping, hard rejection) observes $\alpha + \beta$ and acts as if it were seeing $\alpha$. Whether that is a reasonable thing to do is not a property of the intervention. It is a property of how much of the observation is signal.

Before we can talk about the α/β decomposition, a quick recap on the different importance-sampling quantities your RL framework might be capturing.

Table 1 — Which ratio did you measure?

β is our cross-engine noise, and β′ is a within-engine noise term. β′ exists because of operator differences in batch size; it is typically much smaller than the cross-engine difference, but it is still there.

A synchronous training stack that decomposes hands its trust region a same-engine ratio that genuinely starts at 1 (row 1) and hands its mismatch machinery pure noise (row 3). A stack that does not decompose trains on row 4 and hands its machinery a sum.

The original TIS method proposed to separate the combined ratio into the PPO factor (row 1) and the training–inference factor (row 3), so the trust region operates on a ratio that genuinely starts at 1 [1]. That factoring is precise, but row 3 makes this post’s point vivid: with identical weights on both sides, the training–inference ratio contains no policy movement at all. It is pure β by construction. The field isolated the noise into its own factor, correctly. Then it handed that isolated noise to importance-sampling machinery, as if it were a distribution shift to be corrected rather than a measurement error to be budgeted.

Table 2 — Where each component comes from, and what it scales with.

Before reaching for any correction mechanism, shrink what you can. Every β component you eliminate cleans the ratio for every downstream consumer: the trust region, the staleness correction, your monitoring. It does so without touching α, and unlike every treatment in §4, removal introduces no bias and deletes no signal. The toolbox is its own post [4]: specified divergences reconstructed exactly or honored by construction, discrete selections replayed or pinned, arithmetic path differences unified away, each at its own price.

What survives the toolbox is the subject of this post. Define, per token population or per confidence bin,

The rest of the post:

• §2: the structure of the residual noise, and why it lands exactly where RL can least afford it.
• §3: the procedure for measuring your own split.
• §4: what the measurement licenses, which weights to trust, and where to draw the bounds.

2. The Structure of the Noise

Suppose the residual β were well-behaved: small, symmetric, spread evenly across tokens. Then the story could end here, with a little uniform jitter on every ratio, washing out over any batch, safely ignorable. The measured residual is the opposite on every count. This section presents the general case first, meaning what β looks like when nothing is drifting. It then layers on what the corrections themselves, and training, do to that shape.

Concentrated, Not Spread

To see β alone, freeze everything that could produce α: identical weights on both engines, no updates between sampling and scoring.

Table 3 — The frozen-weights $\lvert\Delta\log\pi\rvert$ distribution (identical weights on both engines).¹

The mean is the least representative number here. The median sits two hundred times below it, and 48% of tokens agree to better than 10⁻⁴. Meanwhile the worst token is off by 7 nats. Ranked by $\lvert\Delta\rvert$, the top 1% of tokens carry 30% of the total mass, and the top 10% carry 80%.

That shape rules out the two boring explanations at a glance. Uniform numerical noise would drag the median up alongside the mean; a systematic engine offset would move the median off zero. Neither happens. β is not a property of the stack applied evenly to every token. It is a property of a small minority of tokens, and everything downstream depends on which minority.

The Token-Wise Amplifier

The minority is not random. Model the accumulated numerics as a small perturbation $\varepsilon_j$, variance $\sigma^2$, on each logit. For the sampled token, $\Delta \log \pi_t = \varepsilon_t - \sum_j p_j \varepsilon_j$, which gives

At a confident token, $p_t \to 1$ and $\lVert p\rVert^2 \to 1$: the variance collapses to zero, so the noise is invisible, however large σ is. At a high-entropy token, $p_t$ is small and the mass is spread so $\lVert p\rVert^2$ is small too: the variance approaches the full $\sigma^2$, and the noise is expressed at full strength. The upstream noise is the same everywhere; the softmax gates how loudly it appears, and the gate is the model’s own confidence.²

The formula holds at scale. Binned by $p_t$, our mean $\lvert\Delta\rvert$ runs monotonically from 0.00025 at $p_t > 0.99$ to 0.284 at $p_t < 0.1$, a factor of ~1100 across the confidence axis. The 8.6% of tokens with $p_t < 0.3$ carry 54% of all $\Delta$ mass; the correlation between confidence and $\lvert\Delta\rvert$ is −0.50. And the non-uniformity is not particular to our stack. ByteDance’s collapse post-mortem scattered millions of per-token trainer-minus-sampler differences and reports the same shape: “the mismatch was not uniform” [5]. Divergence was most severe on tokens the inference engine assigned low probability, and the extreme deltas occurred almost exclusively as the sampler’s probability approached zero.

Figure 1 — The gate, measured. Mean $\lvert\Delta\log\pi\rvert$ per confidence bin under our deployed correction config (the weight-matched calibration of §3). The frozen-weights curve is β alone: same engines, same upstream noise in every bin, yet ~120× more expressed at the forks than at the confident end — the softmax gate. The live curve carries movement plus noise; the gap between the curves is what §3’s subtraction extracts.

And the tail is long in absolute terms: per-token β reaches 7 nats even at frozen weights, with noise alone producing importance ratios of $e^7 \approx 1100\times$.

The Layer-Wise Amplifier

The softmax gate explains where β is expressed, but not why σ, the accumulated logit-level perturbation, is large enough to matter in the first place. The per-operation numerical differences between two BF16 engines are tiny, fractions of a percent per layer. Why do the logits see enough noise to produce nat-scale excursions at all?

Our first answer was a hunt for a culprit. We probed one operator at a time: the output head, reduction orders, attention paths, expert compute. Every probe came back null. Each contributes only the noise you would expect from BF16 arithmetic, far too small to explain the floor. No kernel owned the mean.

What we found instead was amplification in transit. A 1% perturbation injected at layer 0 grows to 24% by mid-network: ~35× compounding, per-layer gain ≈ 1.09, saturating where the branch-input RMSNorm bounds further relative divergence. The floor is distributed, small-everywhere noise made large by the residual stream it travels through.

Why would a trained network amplify perturbations? Our conjecture is that it must: forward noise gain is the price of gradient flow. Linearize the residual stream, and a perturbation injected at layer $l$ reaches the output through $\prod_k (I + J_k)$. The gradient reaches layer $l$ through the transpose of the same product. The singular values are identical, so forward noise gain and backward gradient gain are one quantity, not two.³ A network with “stable outputs” would, by the same operator, have vanishing gradients.

Pre-norm is no special villain here. Any structure that keeps gradients alive across depth carries positive forward gain, and that is what training selects for: dynamical isometry, read in reverse. The circumstantial evidence lines up. Pre-norm holds gradients roughly constant across depth [6], and its measured forward gain compounds to match. Post-norm’s vanishing gradients dually predict noise attenuation, so the architecture the field abandoned for trainability is the one that would have suppressed the mismatch.

So the amplifier is two-stage, and it starts from per-layer numerical noise that is individually negligible:

1. Residual-stream compounding: ~35× on our depth, and irreducible while the network remains trainable if the duality above holds.
2. The softmax gate, $\sigma^2(1 - 2p_t + \lVert p\rVert^2)$, which aims the result at the high-entropy tokens.

What comes out is the heavy-tailed per-token β measured above. Neither stage is a bug, and the hunt’s null was the tell: there was never a culprit to find. The mismatch floor is the forward shadow of gradient flow. That leaves exactly two actionable surfaces: σ itself, through the field guide’s parity and precision levers, and the decision layer that consumes the ratio.⁴ That is the strongest argument that the SNR problem is permanent rather than an engineering accident.

The Cruel Overlap: The Tail Is the Signal

Learning concentrates where the policy is uncertain; the amplifier expresses noise where the policy is uncertain. They are two minorities selected by the same statistic, an overlap built into the softmax by the same $1 - 2p_t + \lVert p\rVert^2$ factor.

Wang et al. measured which tokens matter in RLVR: essentially all of the training signal lives in the high-entropy minority, the “forking tokens,” the ~20% of positions where the reasoning could go more than one way [7]. Policy gradients restricted to that minority match or beat the full gradient; training on the confident majority alone degrades the model.

On our runs the fork set (bottom 20% by sampler confidence) carries 81% of all mismatch mass. Of everything a $[0.5, 2]$ rejection band touches, 89% is a fork token. β is not merely correlated with the forks; it is located on them. A rejection band keyed on the ratio is therefore a near-pure fork-token deleter. Mismatch does not merely coexist with the learning signal; it is aimed at it.

Sorting tens of millions of per-token diffs from our training runs by $\lvert\Delta\rvert$ and reading the top of the list, the worst offenders are fork tokens, by inspection: reasoning-step verbs and branch words, all at $p_t < 0.03$:

Table 4 — Worst per-token disagreements in tens of millions of live training tokens.

The headline row is 'submit': the mismatch is corrupting credit assignment on the very action that earns the reward.

Corrections Reshape the Tail

One honest caveat: β-reduction is not per-token monotone. When we enabled top-p mask replay in our runs, the bulk of the distribution snapped into place. 67–70% of positions became exact logprob matches, with median difference → 0. But the high-entropy tail got sharper: mean $\lvert\Delta\rvert$ in the $p_t < 0.1$ bin rose from 0.284 to 0.425. The mechanism is the amplifier. Nucleus renormalization concentrates the kept probability mass, which raises $\lVert p\rVert^2$, the very quantity that gates how loudly residual noise is expressed. The lesson generalizes: corrections reshape the noise they don’t remove, so re-measure after enabling any fix.

Table 5 — The tail through collapse (one agentic run; the frozen column is the replay-OFF calibration, the live columns are the deployed replay-ON stack).

Read the rows against each other. The raw all-positions mean at collapse reads better than healthy (0.068 vs 0.108), while the mean over non-matching tokens reads twice as bad. The exact-match bulk dilutes the average precisely when the tail runs away, so a dashboard averaging everything sees improvement while the run dies.

The escalation is also not where a maximum-watcher would look. The worst single token was always terrible (7 nats even frozen), and the max barely moves into collapse. What migrates is p95 and p99, and the fork bin as a population: a full nat of disagreement on average across every token in the bin, holding nearly half the total mass. Collapse is not the mismatch getting uniformly worse; collapse is the high-entropy tail running away.

The Tail Moves with the Policy

Entropy. β expression is policy-conditioned, not just stack-conditioned: the gate $1 - 2p_t + \lVert p\rVert^2$ belongs to the current policy, so expressed mismatch moves whenever policy entropy moves, at fixed engine fidelity. A sharpening policy reads as improving aggregate $\lvert\Delta\rvert$ even while it dies by other means. Conversely, every entropy-raising intervention (harder data, an entropy target, a mid-run data refresh) mechanically thickens the expressed tail and the out-of-band mass. Under a rejection band, healthy exploration funds its own deletion. Entropy management and mismatch management are coupled through the gate; a stack that tunes them independently will be surprised by exactly this coupling.

Context length. β accumulates along the trajectory, so long-context workloads sit higher on every curve above. That is one reason the literature’s verdicts diverge along the context axis (§1). The magnitude also travels with the prompt population rather than the engine: vLLM and SGLang stacks both show the entropy-gated shape, varying ~2× across populations.

Agentic tasks. “Longer contexts” undersells them. Tool outputs are text the model never generated. They carry no α (masked from the loss) and no replayable sampler distribution, so their effect is indirect. They push the model into unfamiliar contexts, where its next tokens come out at elevated entropy. That is precisely the amplifier’s high-gain region. Agentic workloads raise β without adding a single new noise source: they steer the token population into the regime where the existing noise is expressed loudest. And 'submit' above stops looking like an accident of sampling: the tool boundary is where unfamiliar context, high entropy, and reward-critical actions all meet.

That is the noise: heavy-tailed, architecturally amplified, aimed at the forks, reshaped by your own corrections, and coupled to your policy’s entropy. What none of this tells you is the split on your stack: how much of your ratio’s variance is this noise, and how much is the movement your machinery exists to handle. That is a measurement, and it is cheaper than you think.

3. Measurements and Logging

Measuring SNR

The prerequisites, per token:

logp_sampler[t] : recorded by the rollout engine at sampling time
logp_trainer[t] : recomputed by the training engine
p_t             : trainer-side confidence of the sampled token (binning key)

Step 0 — bookkeeping sanity. The uncapped mean ratio over any large batch must pin at 1, regardless of policy movement (the identity below). If it does not: stop — you have an evaluation asymmetry, unfaithful logprobs, or a support bug.

assert mean( exp(logp_trainer - logp_sampler) ) ≈ 1    # uncapped, all tokens

For faithful per-token behavior log-probabilities, the uncapped mean ratio satisfies

On 27–31M tokens from our agentic runs it measured 1.00011 and 1.00008. The identity holds live, to four decimals.

Step 1 — calibration pass: β alone. Freeze one checkpoint θ*, load the same θ* in both engines, and generate with your deployed sampling and correction config ON (replay, masks, temperature). Corrections reshape β, so calibrate the config you actually run: enabling mask replay alone moved our fork-bin β ~1.5× (§2). Calibrate at the weights of the live window, not at base. β grows with training: our fork-bin Var(β) at the trained checkpoint measured ~5× its base-weight value, and a base-weight calibration flattered our SNR 2–4× before we caught it.

Δ_cal[t]  = logp_trainer[t] - logp_sampler[t]      # α ≡ 0: same weights
Var_β[b]  = var( Δ_cal[t] for t with p_t in bin b )

Step 2 — live pass: α + β together. The same statistic, over ordinary training batches. Weights now differ by whatever your config produces (staleness, updates).

Δ_live[t] = logp_trainer[t] - logp_sampler[t]
Var_r[b]  = var( Δ_live[t] for t with p_t in bin b )

Step 3 — the subtraction. It assumes α and β are roughly independent within a bin, which is why the binning comes first: both components ride the entropy axis, and conditioning on the bin removes most of the shared dependence. The max(·, 0) is not decoration either. A reading pinned at ≈0 is not a failed measurement; it is what a truly isolated engine-mismatch factor returns (the justification for the last row of §4’s table). Do not assume it is what your folded ratio will return. Ours didn’t, as below.

Var_α[b]  = max( Var_r[b] - Var_β[b], 0 )          # signal variance
SNR[b]    = Var_α[b] / Var_β[b]
λ[b]      = SNR[b] / (1 + SNR[b])                  # shrinkage lookup (§4)

Refresh. β moves as training moves (§2), so redo Step 1 on a checkpoint cadence. If Var_α grows, whether from a config change, more updates per rollout, or looser staleness, then λ rises and ratio-conditioning re-engages on its own. The correction adapts to its own justification.

Run on our own stack, the procedure returns Table 6. The setup: one frozen calibration batch generated under the deployed config (mask replay and routing replay on), recomputed on the trainer at the same trained checkpoint the live window sits at. That makes α ≡ 0 by construction, with an identity check of 1.0002. We compare it against live variance from four healthy steps of an agentic run (39M masked tokens, identity 1.0002).

Table 6 — The procedure, run on our own stack. Weight-matched frozen calibration against four healthy live steps: per-confidence-bin variance split, SNR, and shrinkage λ.

Three readings:

• Every bin is signal-dominated. SNR runs roughly 4–6 across the axis, dipping to 3.8 in the 0.90–0.99 bin and reaching 12 in the near-deterministic top bin, where both variances are vanishingly small. The folded ratio is mostly α.
• Var_α is fork-concentrated, at 0.574 for the forks against 0.0001 at the confident block.
• λ ≈ 0.82–0.87 everywhere. The prescription for this ratio is mostly-trust: temper, bound the tails, never delete. That is what the clamp approximates, and what §4’s harm ordering finds from the other direction.

Reading Existing Logs

Apart from the SNR measurement, most RL frameworks have built-in statistics that can serve as proxies, but they are often conflated.

Table 7 — Existing dials, and the quantity each one actually measures.

• The capped mean is the dial we ourselves misread: ours drifted 0.994 → 0.989 and convinced us of a systematic trainer-side depression, while the uncapped mean sat pinned at 1 the whole time.
• Mean $\lvert\Delta\rvert$ is the dial with no floor under it. A batch with more high-entropy tokens reads “worse” at identical engine fidelity, so an apparent improvement can be the policy sharpening, and an apparent regression can be harder prompts.
• ESS is, in practice, a sequence-level dial. It only fires when sequence-level IS is computed and whole sequences get clipped, the regime where default settings clip ~20% of trajectories. Token-level IS at a healthy clip rate (<1%) never triggers it: with 99% of weights near 1, ESS stays pinned at ~N regardless of what the tail is doing.

4. What the Measurement Licenses

The Criterion

Table 8 — The criterion. SNR of the measured ratio → correct handling.

A run that survives under standard TIS may still be paying an unmeasured performance tax. Misweighted or quietly deleted fork gradients degrade before they destroy, and degradation leaves no alarm on any dial.

Treatment: Never Delete on a Fat-Tailed Ratio

The moderate row’s verdict rests on our own matched pair: same codebase, same correction stack, same two-sided band. One thing changed, delete versus clamp at the boundary. The rejection arm collapsed; the clamp arm held.

The mechanism is §2’s cruel overlap closing into a loop. 89% of what the band touches is a fork token. Deleting fork gradients degrades the policy exactly where it learns, uncertainty grows, the tail thickens, and the band deletes more. We measured it in the act: deleted mass on the rejection arm escalated 4.3× across the collapse window (0.8% → 3.6% of tokens), accelerating through the crash. The clamp arm rode the same tail-thickening at 2.4× over a longer window and never collapsed.

The verdict replicates beyond our stack:

• CISPO proposed the clamp [10].
• ScaleRL’s controlled recipe study found the CISPO-style clamp the best-performing loss in its ablation, beating clipping-style objectives on asymptotic pass rate, and adopted it in its final recipe [11].
• XoRL found the same clamp objective the best objective-level mitigation in its parity study, well above unclipped importance sampling, though still short of eliminating the mismatch outright [12].

The harm ordering on a fat-tailed folded ratio, worst first: reject > trust unbounded > clamp ≥ shrink. When the out-of-band mass is thin, rejection deletes almost nothing and beats trusting raw ratios. That is how our non-agentic runs were saved at long context. As the tail thickens, the same operation turns self-amplifying: the agentic rejection arm collapsed at an escalating 1–3.6%.

Sidedness: Two-Sided Because β Is There

The original prescriptions were one-sided. PPO’s clip caps only over-correction: bound the direction the optimizer would exploit, leave the rest alone. Decoupled PPO inherits exactly that shape for its trust region [3], and the original TIS cap was one-sided too [1]. That is the geometry you choose when you believe your ratio is signal.

Practice landed elsewhere. Our own runs required two-sided bounds for stability, and the reports agree:

• GLM-5’s long-context recipe bounds the residual two-sidedly [13].
• Kimi K2.5 introduces its token-level band with the explicit note that, unlike PPO clipping, it binds “regardless of the sign of the advantages,” targeting “off-policy divergence amplified by discrepancies between training and inference frameworks” [14].
• K3 inherits the objective, and the report calls the band essential precisely for long-horizon tool use. That is the workload §2 says steers tokens into the amplifier’s high-gain region.

The decomposition explains the migration. One-sided bounds are the right geometry for signal, which is pessimism with respect to the objective. Two-sided bounds are the right geometry for noise. β is symmetric in log space, and censoring one tail of a symmetric contamination converts mean-zero jitter into systematic, noise-correlated bias.

Worse, sign-conditioned clipping leaves the negative-advantage, inflated-ratio quadrant unbounded. That is the exact cell where a β excursion on a fork token lands at full weight. Tokens there never register as “clipped,” so the clip-rate dial certifies health while the open quadrant does the damage. On SAO’s stack, that configuration collapses within ~90 steps with the dial near zero, while their two-sided mask holds for a thousand [15]. The shape of your bound is a statement about what you believe your ratio is. Every report that found the correction “must be two-sided” is confessing, in geometry, that its ratio carries a directionless component.

For a mixed ratio the two prescriptions compose rather than conflict. In-band weights come from the dominant signal (trust the weight, λ ≈ 0.85), and bound geometry comes from the noise (two sides). The composition typically lands on an asymmetric two-sided cap.

The moment the ratio carries α, the edges do different jobs. Genuine movement skews upward in ratio space, because the policy concentrates mass on learned preferences, while downward excursions self-attenuate. That leaves the lower edge mostly policing β’s tail, so the suitable band carries the more tolerant upper endpoint. It is our reading of our own agentic case, and likely of SAO’s: their disclosed math band $[0.7, 6.0]$ is −0.36/+1.79 in log space, nowhere near symmetric [15]. Where exactly the two edges go is a calibration, not a convention. The procedure follows.

Placement: Derive the Bounds, and Refresh Them

Two-sided and clamped, then. But where do the sides go? The band has three jobs, and each derives from its own dial, per confidence bin:

• Noise exclusion: the bounds must sit outside β’s bulk, so take the frozen calibration’s empirical tail quantiles.⁵
• Signal coverage: the bounds must pass genuine movement, so cover ±2σ_α from the live−frozen subtraction.
• Variance guard: cap the bound where the clamped-weight effective sample size would fall through its floor (measured non-binding on our runs).

Combine as $u = \min(\max(u_{\text{noise}}, u_{\text{signal}}), u_{\text{ESS}})$, and separately for the lower edge from the lower-side quantities.

Figure 2 — The three-job rule, on two bins of one calibration. Every interval is a measured quantity (one checkpoint’s start-bound derivation). At the forks, noise exclusion binds. β’s empirical bulk reaches past the signal-coverage interval, so the derived band takes the outer envelope: $[0.45, 2.39]$, asymmetric by construction. The static $[0.5, 2.0]$ sits inside it, making noise decisions. At the confident bin the same rule returns $[0.91, 1.08]$, signal-bound at the top, and there the static band is an order of magnitude too loose in log space.

Run on our own stack, the derivation indicted the static $[0.5, 2.0]$ twice over: the fork band sat inside β’s bulk, while the confident bins were an order of magnitude too loose (Figure 2). The derived bounds also age, with the fork bound growing 2.4× in 25 iterations, so they are re-derived at checkpoint cadence and never extrapolated. And when signal coverage demands bounds the variance guard forbids, that is a diagnosis rather than a tuning dilemma. The ratio carries more movement than a low-variance correction can absorb: reduce the staleness or factor the ratio, don’t split the difference on the band.

5. Conclusion

Five takeaways:

• Know which ratio your machinery holds. “The” importance ratio is several different objects, and your bookkeeping fixes the α/β composition before any statistic is computed. A known staleness level is not a known variance split. Only the subtraction is.
• The noise has structure. β is heavy-tailed, gated by the softmax onto exactly the fork tokens where the learning signal lives, and moved by your corrections, your policy’s entropy, and your workload. No ratio threshold separates it from genuine movement.
• Measure, don’t assume. Two passes and a subtraction give the per-bin split, and the identity check gates the exercise (it certifies bookkeeping, never on-policyness). We assumed our folded ratio was mostly noise and measured mostly signal. Construction arguments locate you less reliably than measurements do.
• SNR sets the weight policy; tail mass sets the boundary policy. Trust the weight in proportion to λ, and never let a fat-tailed ratio serve as a deletion key. Clamp instead. The bounds themselves are derived quantities: two-sided because of β, asymmetric because of α, calibrated from the same measurement, and refreshed on cadence because every floor in this problem moves.
• Every “fine” is a floor, not a ceiling. The evidence behind the field’s verdicts, ours included, discriminates stability rather than optimality. A surviving run can still be paying a tax no dial reports.

So the question the literature has been asking — is TIS useful? does clipping help? one-sided or two? — was never a yes/no question about an algorithm. It is a measurement: of the ratio your bookkeeping produces, on your stack, under your workload, refreshed as training moves it. Verdicts age with their regimes. Measurements are what carry over.

Appendix: The Field’s Machinery, on Four Axes

The deployed corrections become legible on a four-axis grid:

• Bookkeeping: which ratio the machinery holds.
• Statistic: which function of that ratio is thresholded.
• Calibration: whether the threshold is token-local or batch-coupled.
• Treatment: what happens at the boundary.

Most named methods innovate on one axis while inheriting defaults on the rest.

Table 9 — The field’s machinery, placed on the grid. Each method’s innovated axis in bold; everything else is inherited default.

Two readings of the grid:

• TV-clip’s statistic has an SNR-relevant side effect. Multiplying by $\mu_t$ is the softmax gate’s own confidence axis, so a TV-keyed bound is structurally looser on fork tokens exactly where a raw-ratio bound is tightest.
• The batch-adaptive members share one blind spot. Their statistics are computed on the folded ratio, so they adapt to $\mathrm{Var}(\alpha) + \mathrm{Var}(\beta)$ and are composition-blind. A batch spread wide by genuine movement and one spread wide by noise gate identically, and everything in this post says they deserve opposite handling.

The criterion is the completion of the calibration move: calibrate to the spread’s composition, not its size. (VCPO’s step scaling is direction-preserving, the mildest of the damping family, but it is keyed on an on-policy floor that our measurements say goes stale within tens of iterations. Recalibrate it weight-matched, like every other floor in this post.)

For a pure-β ratio the correct importance weight is $e^{0} = 1$. TIS’s $\min(r, C)$ on that ratio is a reweighting and a censoring welded together. The reweighting multiplies each gradient by a mean-one lottery draw, so the entire benefit, if any, must live in the censoring half. The cost side is measured: TRL’s phantom clipping left raw gradient cosine above 0.95 and the clip machinery collapsed it to 0.55 [9]. The mismatch barely bends the gradient; the boundary, crossed by noise on tokens whose policy never moved, is what breaks it.

References

[1] Yao, F., Liu, L., Zhang, D., et al. “Your Efficient RL Framework Secretly Brings You Off-Policy RL Training.” 2025. Flash-RL.

[2] Xi, H., Ruan, C., Liao, P., et al. “Jet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flow.” NVIDIA / MIT / UC Berkeley, January 2026.

[3] Off-Policy Corrections in LLM RL Training. March 2026.

[4] A Field Guide to Training–Inference Corrections. September 2026.

[5] Liu, J., Li, Y., Fu, Y., Wang, J., Liu, Q., Shen, Y. “When Speed Kills Stability: Demystifying RL Collapse from the Training-Inference Mismatch.” ByteDance, September 2025.

[6] Xiong, R., Yang, Y., He, D., et al. “On Layer Normalization in the Transformer Architecture.” ICML 2020.

[7] Wang, S., Yu, L., Gao, C., et al. “Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning.” Qwen Team / LeapLab, June 2025.

[8] Large, T., Liu, Y., Huh, M., et al. “Training Transformers with Enforced Lipschitz Bounds.” July 2025.

[9] aminediroHF. “Defeating the trainer-generator precision mismatch in TRL.” 2026.

[10] MiniMax. “MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention.” June 2025.

[11] Khatri, D., Madaan, L., Tiwari, R., et al. “The Art of Scaling Reinforcement Learning Compute for LLMs.” Meta, October 2025.

[12] Panda, A. “0 train-infer mismatch for Open-weight MoE RL in Open-source code.” TogetherAI, 2026. XoRL.

[13] GLM-5 Team, Zhipu AI & Tsinghua. “GLM-5: from Vibe Coding to Agentic Engineering.” February 2026.

[14] Kimi Team. “Kimi K2.5: Visual Agentic Intelligence.” February 2026, §4.4.2.

[15] Hou, Z., Li, Y., Tang, J., Dong, Y. “Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning.” Tsinghua / Z.AI, July 2026.

[16] Qi, P., Zhou, X., Liu, Z., et al. “Rethinking the Trust Region in LLM Reinforcement Learning.” ICML 2026.

[17] Yang, J., Shi, Y., Li, Z., et al. “Staleness-Adaptive Trust Region for Asynchronous RL.” Tencent, July 2026.

[18] Zheng, H., Zhao, J., Chen, B. “Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?” CMU / Meta AI, ICLR 2026.

[19] Huang, L. J., Zhang, Z., Hu, Q., Yang, S., Han, S. “Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs.” MIT / NVIDIA, February 2026.