Off-Policy Corrections in LLM RL Training
Published:
A unified treatment of the five sources of distribution mismatch in LLM reinforcement learning and their corrections.
Posts by Tags
PPO
RL
Off-Policy Corrections in LLM RL Training
Published:
A unified treatment of the five sources of distribution mismatch in LLM reinforcement learning and their corrections.
Understanding Length Dynamics in RL Training
Published:
An empirical investigation into what drives output length growth during RL training, revealing that dataset difficulty composition is the primary driver behind the ‘overthinking’ phenomenon.
advantage-estimation
Implementing Training-Free Process Rewards in VeRL
Published:
A training-free approach to process rewards: estimate V(prefix) via log-probability, compute marginal utility across episodes. Plus VeRL implementation pitfalls to avoid.
chain-of-thought
Understanding Length Dynamics in RL Training
Published:
An empirical investigation into what drives output length growth during RL training, revealing that dataset difficulty composition is the primary driver behind the ‘overthinking’ phenomenon.
evaluation
What’s in Pass@K?
Published:
Pass@k is ubiquitous in evaluating reasoning models, but the metric is more subtle than it appears. Computing it correctly requires the unbiased estimator, and the nonlinearity of pass@k means it effectively upweights hard problems compared to pass@1.
importance sampling
Off-Policy Corrections in LLM RL Training
Published:
A unified treatment of the five sources of distribution mismatch in LLM reinforcement learning and their corrections.
knowledge-distillation
Implementing On-Policy Distillation: Lessons from Building OPD in VeRL
Published:
On-policy distillation integrates teacher guidance into RL training, but the implementation is full of silent failures. This post documents the architecture, pitfalls, and design choices from building OPD in VeRL.
length-dynamics
Understanding Length Dynamics in RL Training
Published:
An empirical investigation into what drives output length growth during RL training, revealing that dataset difficulty composition is the primary driver behind the ‘overthinking’ phenomenon.
off-policy
Off-Policy Corrections in LLM RL Training
Published:
A unified treatment of the five sources of distribution mismatch in LLM reinforcement learning and their corrections.
on-policy-distillation
Implementing On-Policy Distillation: Lessons from Building OPD in VeRL
Published:
On-policy distillation integrates teacher guidance into RL training, but the implementation is full of silent failures. This post documents the architecture, pitfalls, and design choices from building OPD in VeRL.
post-training
Off-Policy Corrections in LLM RL Training
Published:
A unified treatment of the five sources of distribution mismatch in LLM reinforcement learning and their corrections.
process-rewards
Implementing Training-Free Process Rewards in VeRL
Published:
A training-free approach to process rewards: estimate V(prefix) via log-probability, compute marginal utility across episodes. Plus VeRL implementation pitfalls to avoid.
reasoning
What’s in Pass@K?
Published:
Pass@k is ubiquitous in evaluating reasoning models, but the metric is more subtle than it appears. Computing it correctly requires the unbiased estimator, and the nonlinearity of pass@k means it effectively upweights hard problems compared to pass@1.
Understanding Length Dynamics in RL Training
Published:
An empirical investigation into what drives output length growth during RL training, revealing that dataset difficulty composition is the primary driver behind the ‘overthinking’ phenomenon.
reinforcement-learning
Implementing Training-Free Process Rewards in VeRL
Published:
A training-free approach to process rewards: estimate V(prefix) via log-probability, compute marginal utility across episodes. Plus VeRL implementation pitfalls to avoid.
Implementing On-Policy Distillation: Lessons from Building OPD in VeRL
Published:
On-policy distillation integrates teacher guidance into RL training, but the implementation is full of silent failures. This post documents the architecture, pitfalls, and design choices from building OPD in VeRL.
verl
Implementing Training-Free Process Rewards in VeRL
Published:
A training-free approach to process rewards: estimate V(prefix) via log-probability, compute marginal utility across episodes. Plus VeRL implementation pitfalls to avoid.
Implementing On-Policy Distillation: Lessons from Building OPD in VeRL
Published:
On-policy distillation integrates teacher guidance into RL training, but the implementation is full of silent failures. This post documents the architecture, pitfalls, and design choices from building OPD in VeRL.