A reward ceiling for RL alignment, computed before training

The most reward that reinforcement learning can add under a KL constraint is computable before training from base-model samples alone, and on that ceiling best-of-N reached over 98% while PPO and GRPO stayed at 77% and 75% on the safety task ^[1]. An Apple team derived the optimal reward gain of KL-regularized alignment in closed form as a Jeffreys divergence and rewrote it as a covariance under the base model so it can be estimated before training ^[1]. The same framework proves why reward hacking happens and why reward model ensembles help ^[1].

This article reads the body, theorems, and appendix tables of the paper posted on arXiv and checks the figures against them ^[1]. The models and datasets are public assets, but we have not reproduced the experiments, and we have no relationship with the authors.

Where is the limit of RL alignment?

The reward an optimally aligned model gains over the base model equals the KL penalty λ times the Jeffreys divergence between the two models ^[1]. The starting point is the KL-regularized problem that RLHF-style methods share. A reward model $r$ scores an answer $y$, and deviating from the base model ${\pi }_{\text{base}}$ is penalized through the KL divergence ^[1].

This problem has a known closed-form solution: the base model's probability multiplied by the exponential of the reward and renormalized ^[1].

Here $\lambda$ is the strength of the KL penalty and $Z$ is the normalizer. How much reward this solution gains was known only as a bound. Gao et al. reported an empirical rule that the gain grows roughly with the square root of KL ^[2], and Mroueh and Nitsure proved that square-root expression is an upper bound under a sub-Gaussian reward ^[3]. Theorem 1 of this paper gives an equality with no assumption ^[1].

Here $\mathrm{JD}$ is the Jeffreys divergence, the sum of the forward and reverse KL. The gain is "how far the model moved from the base," measured in both directions, times the penalty λ, and a one-directional square root of KL does not describe it ^[1].

Can the reward gain from alignment be computed before training?

Yes. The right-hand side of Theorem 1 needs the aligned model, but the same quantity rewritten as a covariance under the base model needs no aligned model at all ^[1]. Theorem 2 separates the reward $r^{\prime }$ used for evaluation from the reward $r$ used for training so one expression covers both cases ^[1].

Here ${\mathrm{Cov}}_{{\pi }_{\text{base}}}$ is the covariance over answers sampled from the base model, and the second argument is the exponential weight each answer receives. Optimal alignment is a reweighting that favors the higher-reward answers the base model can already produce, so base samples plus reward scores are enough to see how much reweighting can add.

A Monte Carlo estimator for this covariance (Definition 1) and a KL estimator built from the same ingredients (Definition 2) follow, and both are proven to converge as the number of samples grows ^[1]. In the paper's experiments the reward-gain estimate converged within about 1,000 base samples per prompt, and convergence was slower for small λ because the exponential weight inflates variance ^[1]. Sweeping λ from 0.01 to 10 yields the maximum reward at every KL budget, the whole Pareto frontier, before training ^[1].

Figure 1. The most reward alignment can add is computed before training as a covariance under base-model samples, and the estimate converges as samples grow ^[1].

Why do PPO and GRPO get less reward than best-of-N?

On both tasks best-of-N sat almost on the ceiling curve, while PPO and GRPO stayed below it at every KL budget ^[1]. The experiments cover safety alignment and summarization. The safety task starts from Zephyr-7B (Mistral-7B supervised fine-tuned on UltraChat) with a Zephyr-7B-based reward model trained on PKU-SafeRLHF preferences ^[1]. The summarization task starts from Pythia-1B supervised fine-tuned on Reddit TL;DR with a Pythia-1B reward model trained on the same preferences ^[1]. The compared methods are PPO, GRPO, and best-of-N; DPO is excluded because it uses no explicit reward model ^[1]. PPO and GRPO went through an extensive hyperparameter sweep and the run with the best KL-reward trade-off is reported, and best-of-N varies N from 1 to 10,000 with its KL computed by the estimator of Beirami et al. ^[1][4].

Table 1. Maximum reward gain of each method within the KL range where the ceiling was computed (up to 9.14 for safety, up to 7.96 for summarization) ^[1]. GRPO and PPO values are from the runs shown in the main figure; for summarization two higher PPO runs were excluded as too noisy ^[1].

Computing ratios from Table 1, best-of-N reaches 98.5% and 99.5% of the ceiling, GRPO 76.6% and 72.9%, and PPO 75.4% and 48.3% ^[1]. The authors read GRPO and PPO staying below the ceiling at every KL budget as a structural limitation of current RL methods, and write that algorithmic improvements are needed to reach the optimum without high inference costs ^[1]. The paper offers no analysis of why gradient-based methods fall short of the optimal reweighting.

Figure 2. On safety alignment best-of-N sits on the dashed ceiling while GRPO and PPO stay below it at every KL budget ^[1].

The appendix sweep shows the same gap. On the safety task, 13 GRPO configurations of β, learning rate, and number of generations produced maximum gains between 1.886 and 2.117, all on one band below the ceiling, and 13 PPO configurations spread from 0.918 to 2.084, all below it ^[1].

Figure 3. Every GRPO hyperparameter configuration collapses onto the same band below the ceiling ^[1].

Why does reward hacking happen and why do ensembles help?

Reward hacking happens because the proxy reward's error enters an exponential divided by the KL penalty λ ^[1]. The reward model $\tilde{r}$ used in training approximates the true reward $r^{\ast }$ that perfectly reflects human preference, and the paper calls the difference $N(x,y)=\tilde{r}(x,y)-r^{\ast }(x,y)$ the preference residual ^[1]. Proposition 3 writes the true-reward gap between a model aligned to the true reward and one aligned to the proxy as a covariance containing an $e^{N(x,y)/\lambda }$ term ^[1]. Shrinking λ to move further from the base amplifies the same error exponentially, which explains the overoptimization curve that Gao et al. observed ^[1][2].

The effect of reward model ensembles is proven in the same framework. Proposition 4 is a Lipschitz-style bound tying the loss from aligning to a proxy to the $L_2$ distance between proxy and true reward, and Theorem 3 shows that averaging n independently trained reward models makes the aligned model's true reward converge to the ideal at a rate inversely proportional to the square root of n ^[1]. That gives a theoretical basis to the ensemble effect Coste et al. reported empirically ^[5]. It assumes each reward model equals the true reward on average, an unbiasedness condition ^[1].

Appendix C also examines how reward models are trained. Most reward models today are trained with the Bradley-Terry loss, which fits only the order of preferences and places no constraint on how much one answer is preferred ^[1]. So even a perfectly minimized loss leaves the preference residual free to grow ^[1]. The authors' example is data where comparisons happen only within separate clusters, such as helpful answers and safe refusals. The score offset between clusters never appears in the loss and can drift arbitrarily, and that offset inflates the exponential term in Proposition 3 ^[1]. Their proposal is reward modeling that also learns the magnitude of preference ^[1].

What are the limits of this study?

The authors state three limits. The results characterize the optimal solution of the KL-regularized objective, so they do not transfer directly to f-divergence regularization, win-rate maximization, or per-token KL ^[1]. The estimators converge slowly for small λ ^[1]. The reward hacking and ensemble analysis assumes proxy rewards are unbiased on average, and systematic bias is not covered ^[1].

We add a few from our reading. The experiments cover two model-dataset pairs, Zephyr-7B and Pythia-1B, on two single-turn tasks, with no multi-turn agent or coding task ^[1]. Reward gain is scored with the same reward model used for training, so sitting on the ceiling is not a claim about human-judged quality. Best-of-N's KL is an estimated value ^[1][4]. PPO and GRPO are the best runs of the authors' sweep, so how the gap behaves under other implementations or larger models is open. It is an arXiv preprint, and peer review is unconfirmed.

How can this result be used?

The first use is measuring the ceiling before training. Sampling about 1,000 answers per prompt from the base model and scoring them gives the maximum reward at each KL budget, and the KL budget needed for a target reward can be read off in reverse ^[1]. In our reading, a low ceiling points to the base model or the reward model as the constraint, not the algorithm.

The second is narrowing down where to look when training stalls. The ceiling can be estimated from base samples, and the gap between an actual run and the ceiling shows up by plotting the run on the same axes ^[1]. The proxy reward's residual cannot be measured directly, since it needs the true reward, but the paper gives its direction, exponential amplification as λ shrinks, and a mitigation, ensembling ^[1].

The third is keeping best-of-N as the reference line. In the paper's experiments best-of-N sat on the ceiling, so a new algorithm's result can be read as its distance from best-of-N ^[1]. The authors leave algorithmic improvement that reaches the optimum without high inference costs as the open task ^[1].