Continual RLVR Post-Training & Verifier Gates
Co-authoring "If-Taught Reasoners Forget How to be Safe" (under review): verifier gates and alignment replay buffers that cut Attack Success Rate across 10+ continual rounds, with Dr. Chhavi Yadav.
Overview
Reasoning models trained with reinforcement learning and verifiable rewards (RLVR) over 10+ continual rounds slowly learn to game their evaluators: verbose rubric matching, fabricated tool outputs, quiet erosion of earlier safety behavior. Co-authoring "If-Taught Reasoners Forget How to be Safe" (under review), quantifying catastrophic safety drift and engineering a dual-mechanism mitigation: a safety-aware verifier and an alignment replay buffer.
What was engineered
- Deterministic trajectory verifier gates: sandboxed Python AST checks, tool-call schema validation, and state invariants that run server-side, away from the policy.
- Prioritized alignment replay buffers: verified trajectories are ranked and replayed so safety behavior persists across continual rounds.
- Cryptographically isolated holdout gates: checkpoints roll back automatically when base task completion or safety benchmarks degrade.
- Multi-turn RL framework: quantifying dynamic social engineering and alignment faking, evaluating how policies mirror user personas to exploit RLVR against a secondary LLM User Simulator.