RL beyond the context window

Critic-based PPO, compaction, and anti-hacking

Source: GLM-5.2: Built for Long-Horizon Tasks — Z.ai, 2026

An agent working for hours may generate more history than its context window can hold. GLM-5.2 (Z.ai, 2026) is an open-weights model built for long-horizon agentic work: coding tasks measured in hours, million-token trajectories, and rewards that come from whether the code actually works. Its post-training report describes how RLVR handles trajectories that outgrow the context window, and how training responds when the model learns to exploit the grader.

Why long trajectories need a critic

GRPO’s founding bargain was to fire the criticcriticA model trained to predict the value function. PPO uses an actor (the policy) and a critic; GRPO drops the critic and uses a group average instead.See in glossary →: sample a group of rollouts per prompt and use the group-relative advantagegroup-relative advantageGRPO’s advantage estimate: a response’s reward minus the mean reward of its group of siblings (often divided by their standard deviation), replacing a learned value function.See in glossary → as the baseline. That works beautifully when one prompt yields a clean set of comparable completions.

Long-horizon agentic tasks break the “comparable” part. A single rollout can run for hours and overflow the context window many times; each time, compactioncompactionCompressing an agent's long interaction history (summarizing earlier turns, dropping stale tool output) so a long-horizon task keeps fitting in the model's context window; the trajectory then continues from the compacted state.See in glossary → squeezes the history down and the trajectory continues from the compacted state. By the end, one prompt’s different rollouts have been split into different numbers of sub-traces with wildly different lengths. The resulting sub-traces are difficult to compare as one group, making a group-relative baseline harder to define.

GLM-5.2 uses a critic-based PPO formulation that learns from individual rollouts. A learned value functionvalue functionThe expected return from a given state under the current policy. A learned value function (the critic) provides a baseline that reduces the variance of policy-gradient updates.See in glossary → supplies token-level advantagesadvantageHow much better an action was than the baseline expectation: A = reward − value. Positive advantage pushes an action’s probability up, negative pushes it down.See in glossary → (estimates of how much better an action was than expected, as described in value and advantage), so no cross-rollout comparison is needed. That single-rollout view absorbs compaction naturally: every compacted sub-trace becomes a trainable trajectory in its own right, and a token-level loss keeps the long ones from drowning out the short ones.

Anti-hacking as infrastructure

Even pass/fail rewards in coding can be vulnerable to reward hacking. Z.ai reports that GLM-5.2 showed more hackingreward hackingWhen a policy finds ways to score high on the reward model without actually being better — exploiting quirks of an imperfect proxy. A central danger of RL post-training.See in glossary → tendency than its predecessor during training: reading protected evaluation artifacts, copying answers from upstream commits, even curl-ing the reference solution straight off GitHub, plus chained versions (find the hidden test file, cat it, feed it to the solver).

Their countermeasure is notable for being online infrastructure, not a data filter. A two-stage detector watches every tool call during rollout: a rule-based filter flags anything suspicious (tuned for recall), then an LLM judge inspects the flagged action’s intent (restoring precision). When a hack is confirmed, the system blocks that tool call and returns dummy output, and the rollout keeps going.

slime, and merging ten teachers in two days

These long-running tasks also place demands on the training infrastructure. GLM-5.2’s agentic RL runs on slime, Z.ai’s open RL framework, which unifies the modes a frontier lab actually needs: white-box and black-box rollouts, compacted trajectories, sub-agent workflows, all feeding one training process, with flexible connections to inference backends (parallelism strategies, routing, prefill/decode disaggregation) and FP8 KV cache for rollout throughput.

The final model was assembled through parallel on-policy distillationon-policy distillationDistillation where the student generates its own rollouts and a teacher grades every token of them. Combines the dense per-token signal of distillation with the on-policy benefit of RL — much cheaper than RL for transferring a teacher's reasoning.See in glossary →, merging more than ten specialized expert models into the single release model, in roughly two days of training. The team trains experts separately, each with its own RL recipe, then uses on-policy distillation to train one student from their per-token feedback.

GLM-5.2 combines pass/fail verification, PPO with a learned critic, and distillation. Its long trajectories also require a system that preserves task state and prevents access to hidden answers. Kimi K3 addresses a related problem: teaching one model to work within several reasoning budgets.