Reasoning effort is trainable

Budget-controlled RL, nine teachers, and persistent rollouts

Paper: Kimi K3: Open Frontier Intelligence — Kimi Team, 2026

Inference scaling is usually described as a serving-time choice: let the model think longer when a problem is hard. Kimi K3’s post-training recipe makes a stronger claim. Reasoning effort can itself be a training target. The team trains policies for different domains and budgets, then consolidates them into one model that can operate at low, high, or maximum effort.

That target changes the shape of the RL system. A maximum-effort coding agent may make thousands of tool calls and accumulate millions of context tokens. Its rollout cannot be treated like a long chat completion that must finish before the next update. Training has to preserve the policy’s state, the environment’s state, and the intended effort level across many optimizer iterations.

Three domains times three effort levels

K3 divides RL into three broad domains:

• General tasks: knowledge, reasoning, vision, search, and faithfulness.
• General agents: deep research, long-form writing, and assistant workflows.
• Coding agents: software engineering, kernels, and web development.

For each domain, it trains low-, high-, and max-effort experts: nine policies in total.

The effort control is learned with a per-problem budget. The cold-start model first estimates a baseline token budget $b_0(x)$ for problem $x$. A rollout loses its task reward if its total output exceeds a multiple $\tau b_0(x)$. For ordinary reasoning, that total counts thinking tokens; for agents, it includes reasoning and tool-call arguments across the trajectory.

Training begins with a relatively generous multiplier to produce the max-effort expert, while still preventing unlimited overthinking. The multiplier is then gradually reduced for high and low effort. This is more targeted than one global response-length penalty: a task that naturally needs 20,000 tokens gets a larger allowance than one that normally needs 500.

For non-verifiable work, an agentic generative reward modelgenerative reward modelA reward model that is itself a language model: it writes out a critique or reasons step by step before scoring a response, rather than emitting a single opaque scalar. More accurate and interpretable than a bare scalar head.See in glossary → compares candidates after writing a rubric and scoring each candidate against it. A parallel verbosity rule makes overlong outputs automatically lose the comparison. The same budget idea therefore controls both hidden reasoning and the visible temptation to win a judge by saying more.

Partial rollouts: stop waiting for the longest agent

Synchronous RL normally generates a batch of rollouts, waits for all of them, then updates the policy. Long-horizon agents create an extreme stragglerstragglerThe slowest item in a synchronized batch. Every other worker may sit idle while the system waits for this last rollout or computation to finish.See in glossary → problem: one trajectory can continue for hours after most of the batch has finished.

K3 uses partial rolloutspartial rolloutA trajectory that is paused before completion, carried across policy updates with its model and environment state intact, and resumed in a later RL iteration.See in glossary →. Generation pauses once a chosen fraction of active trajectories completes. Finished groups go to policy optimization; unfinished trajectories enter a queue and resume at the start of a later iteration. One logical episode can therefore span multiple policy updates.

That creates stale data: the trajectory began under an older policy than the one currently being optimized. K3 relies on per-token regularization that constrains each update to a local neighborhood, allowing the learner to tolerate these unusually off-policy fragments. A rollout must therefore preserve enough state to resume after a policy update.

The environment must resume too. K3 uses microVMsmicroVMA lightweight virtual machine: an isolated software environment that behaves like a separate computer.See in glossary →, lightweight virtual machines that isolate each task. They can save just the memory regions changed since the previous snapshot, pause without consuming CPU or memory, and create an exact copy of their state so a judge can inspect it without changing the original. Saved snapshots also allow recovery after an interruption. A personal-assistant task can evolve across simulated days and thousands of tool calls without being reconstructed after every RL iteration.

Train across harnesses, not for one harness

An agent can overfit to a tool schema or context-management convention just as a language model can overfit to a prompt format. K3 represents a harnessagent harnessThe software that provides an agent with tools and manages its interaction with a task.See in glossary →, the software that supplies an agent’s tools and manages its interaction with a task, as configurable modules: tools, prompts, memory, compaction, skills, and subagents. It varies their combinations across RL tasks so the model learns to reason, act, observe, verify, and adapt across different setups.

The training tasks reinforce that loop with external state and verifiers. Examples include professional workflows in mock software-as-a-service (SaaS)SaaSSoftware as a Service — software accessed through a network while its provider operates the underlying application.See in glossary → applications, accessed through a network, visual reasoning where Python crops and transforms an image, kernel optimization rewarded for correctness and speed, and autonomous execution tasks whose hidden verifier checks the final environment rather than the agent’s claim that it finished.

Nine teachers become one policy

K3 combines the nine experts into one released model with multi-teacher on-policy distillationon-policy distillationDistillation where the student generates its own rollouts and a teacher grades every token of them. Combines the dense per-token signal of distillation with the on-policy benefit of RL — much cheaper than RL for transferring a teacher's reasoning.See in glossary →. The student samples a domain and effort level, generates its own rollout, and receives a dense per-token log-ratio reward from the matching teacher. Clipping extreme log ratios stabilizes the signal.

Each teacher specializes in both a domain and a computational budget. The released student learns all nine behaviors and an interface for selecting among the effort levels.

Post-training is also deployment-aware. The expert weights use MXFP4 and their activations MXFP8—four-bit and eight-bit floating-point formats that share a scale within small groups of values—throughout SFT and RL, while sensitive non-expert modules stay at higher precision. Rollout and training use the same quantizationquantizationRepresenting numbers with a smaller set of possible values, usually to reduce storage and speed up computation at the cost of some precision.See in glossary → scheme, the rule for representing values with fewer bits, preventing the policy from being optimized in one numerical environment and sampled in another.

The resulting model learns from specialists trained under different budgets. Persistent environments let those specialists complete long tasks across multiple training updates, and distillation brings their behavior into one model with a selectable reasoning effort.