← All notes

Actor-Critic Methods

July 14, 2026 · Updated July 30, 2026

Actor-critic methods build on policy gradients: alongside the policy (the “actor”), they learn a value function (the “critic”) to judge how good the actor’s actions are, giving a lower-variance learning signal than the raw Monte Carlo returns used in vanilla policy gradient.

In one line: policy gradients observe what was good or bad, then do more of the good stuff; actor-critic learns to estimate what’s good or bad, then does more of the good stuff. Same recipe, just swapping a noisy observation for a neural network’s own estimate of the value function.

Motivation: policy gradient makes inefficient use of data

Even with reward-to-go, which already stops an action from being blamed for rewards that came before it, policy gradient still only has one sampled rollout to learn from. Whatever happens after an action in that one particular trajectory still gets baked into its credit, whether or not it’s representative of what usually happens when the policy takes that action. Good and bad actions inside the same trajectory end up reinforced or punished together.¹

This is sharpest under sparse rewards, where the only nonzero signal arrives once, at the very end of the episode. Two trajectories that make very different amounts of real progress toward the goal, but happen to land on the same final reward, get treated as equally good, or equally useless. The algorithm has no way to see the difference.²

All policy gradient ever knows is that a trajectory succeeded or failed as a whole; it has no notion of how good any specific state or action was on its own. That is exactly the gap a learned critic is built to fill: an estimate of a state’s or action’s expected future reward, averaged over the policy’s actual behavior, rather than one noisy, one-off realization of it.

Preliminaries

• value function $V^\pi(s)$: the future expected reward starting at state $s$ and following policy $\pi$ from there
• Q-function $Q^\pi(s, a)$: the future expected reward starting at state $s$, taking action $a$, then following policy $\pi$ from there

$V$ and $Q$ are related: since $V^\pi(s)$ is the expected future reward under $\pi$‘s own behavior at $s$, it must equal $Q^\pi(s, a)$ averaged over the actions $\pi$ would actually choose at $s$,

$Q^\pi(s, a)$ answers “how good is action $a$,” while $V^\pi(s)$ answers “how good is state $s$, on average, under $\pi$‘s own choices.” Averaging the first over $\pi$‘s action distribution gives the second back.

This gives a natural way to measure how good a specific action is relative to the policy’s own average behavior: the advantage function,

how much better (or worse) it is to take action $a$ than to follow $\pi$‘s own behavior at state $s$.

From reward-to-go to Q: a lower-variance estimator

Recall the reward-to-go estimator,

This update increases the probability of actions that led to large future rewards. But $G_{i,t}$ is only ever computed from one sampled rollout: it’s an estimate of the future reward, not the future reward itself. The same first action can lead to wildly different futures.

This is exactly the high variance from the motivation above, made concrete: $G_{i,t}$ swings with whichever future happened to get sampled, even though the action itself, taking $a_t$ at $s_t$, is the same every time.³

A better estimate averages over every possible future after taking $a_t$ in $s_t$, instead of relying on the one future that happened to get sampled:

This is exactly the $Q$-function defined above: the expected future reward from taking $a_t$ at $s_t$ and then following $\pi$. Substituting $Q^{\pi_\theta}(s_t, a_t)$ for the sampled reward-to-go $G_{i,t}$ gives a new, lower-variance estimator:

Instead of saying “this one rollout earned reward 8,” this says “this action is generally worth about 8”: the update is now driven by an averaged, expected value, rather than one noisy, one-off realization of it.

What the Q-function learns

Instead of trusting a single rollout, the critic accumulates experience across many episodes: every time the policy takes action $a$ at state $s$, it records the return that followed, and averages over all of them.⁴ This average is exactly the Monte Carlo estimate of $Q^\pi(s, a)$ defined above, just computed from many rollouts instead of one.

Suppose that after enough episodes, this average settles at $Q(s, a) = 6.8$. The policy update no longer uses the reward of $2$ that one sampled rollout happened to return; it uses $6.8$, the critic’s running average over everything it has seen.

Q still depends on the state, not just the action

Even $Q^\pi(s, a)$ has a subtle flaw: it mixes together how good the state $s$ already is with how good the specific action $a$ is. In a position that’s already winning, every legal move can look great under $Q$, not because any one move is exceptional, but because the state itself already guarantees a good outcome.⁵

The fix is to stop asking “is this action good?” in absolute terms, and instead ask “is this action better than what I usually do from this state?” That’s exactly what the advantage function $A^\pi(s, a) = Q^\pi(s, a) - V^\pi(s)$ measures: subtracting off the state’s own average value, $V^\pi(s) = \mathbb{E}_{a \sim \pi}[Q^\pi(s, a)]$, removes whatever part of $Q^\pi(s, a)$ was just “the state is good,” leaving only the part attributable to the action itself.⁶

Plugging $A^{\pi_\theta}$ into the gradient in place of $Q^{\pi_\theta}$ gives the actor-critic estimator:

Better estimates of $A^\pi$ lead to less noisy gradients. The actor no longer chases large absolute rewards caused by simply being in a good state; it updates based on whether an action was better or worse than expected from that specific state, a more stable and efficient learning signal.

The actor-critic loop

Fitting $V^\pi$, $Q^\pi$, or $A^\pi$ turns actor-critic into a small extension of the online RL loop: collect a batch of data with the current policy, fit a model to estimate the expected return, then use that fitted estimate to improve the policy.

You estimate value metrics like the State-Value $V^\pi$, Action-Value $Q^\pi$, or the Advantage $A^\pi$ (which is seen in the top equation). This gives you the ground truth weight needed for your gradient update.

In practice, this fitted model is a neural network: it takes a state $s$ as input and outputs an estimate $\hat{V}^\pi(s)$, with its own parameters $\phi$, separate from the policy’s parameters $\theta$.

Training the value network

Fitting $\hat{V}^\pi_\phi(s)$ turns into an ordinary supervised learning problem: for every state $s_{i,t}$ visited in a batch, use the observed return $y_{i,t} = G_{i,t}$, the actual reward-to-go collected from that state onward, as the regression label.

• input: the state $s$
• target: the observed return $y$

The loss is standard mean squared error between the network’s prediction and this target,

training $\phi$ by gradient descent on $L$, exactly like any other regression problem.⁷

Every episode that passes through a given state contributes another noisy sample of the return from there. Training on many of these samples pushes the network’s prediction toward their average, which is exactly $V^\pi(s)$: the expected return under $\pi$, not any single episode’s realization of it.⁸

Bootstrapping: expressing Q with just V

Fitting a network for $V^\pi$ alone, without a separate $Q$-network, is enough. Start from $Q^\pi(s_t, a_t)$ written as a sum of expected future rewards, and pull out the very first one, the reward that arrives immediately after taking $a_t$:

Everything after the first reward is, by definition, exactly $V^\pi(s_{t+1})$: the expected future reward from following $\pi$ starting at the next state. There’s an expectation, $\mathbb{E}[V^\pi(s_{t+1})]$, because after your action, many next states are possible. But in reinforcement learning we only ever experience one next state per rollout: current state, action, observed next state, no branching. So the last line approximates that expectation with the one sampled next state: it plugs in $V^\pi$ evaluated at the $s_{t+1}$ that was actually observed, instead of averaging over all of them.

This is called bootstrapping: using the current value estimate itself to stand in for future rewards, instead of needing a whole separate rollout (or a model of the environment’s dynamics) to know what happens next.

Substituting this into the advantage function, $A^\pi(s_t, a_t) = Q^\pi(s_t, a_t) - V^\pi(s_t)$:

No $Q$-network required: the advantage, which needed $Q^\pi$ a moment ago, can now be computed from $V^\pi$ alone, plus the one reward that was actually observed.

Temporal difference learning: bootstrapped targets

The Monte Carlo target used above, $y_{i,t} = G_{i,t} = \sum_{t'=t}^{T} r_{i,t'}$, has a practical problem: you can’t compute it until the episode is over. If an episode runs for $10{,}000$ steps, the network can’t take a single gradient step until step $10{,}000$.

The fix is the same bootstrapped estimate derived above, now used as the training target itself, instead of just for the advantage:

Instead of waiting for the rest of the episode, the label is available immediately: one observed reward, plus the network’s own current guess at everything after it.⁹

At first, that guess can be quite wrong. But every gradient step improves $\hat{V}^\pi$ a little, which means every subsequent target $y = r + \hat{V}^\pi(s')$ gets a little more accurate too: the network is teaching itself, bootstrapping better and better labels out of its own improving predictions.

The loss function doesn’t change at all, only the target does:

Learning is now much faster, since it no longer waits for an episode to end.

This is called temporal difference (TD) learning because of what’s being compared: the current prediction $\hat{V}^\pi(s)$, against a target built from the next time step, $r + \hat{V}^\pi(s')$. Their difference is the TD error,

If $\delta > 0$, the state turned out better than expected, so $\hat{V}^\pi(s)$ should increase. If $\delta < 0$, it turned out worse than expected, so $\hat{V}^\pi(s)$ should decrease.

Bias vs. variance: Monte Carlo vs. TD

The Monte Carlo target from training the value network and the bootstrapped TD target above sit on opposite ends of a classic tradeoff.

The Monte Carlo target, $y = G_{i,t}$, uses the actual future return realized in that one sampled episode. It’s unbiased: averaged over enough episodes, it converges to the true $V^\pi(s)$. But it’s noisy, since a single episode can be lucky or unlucky, so the label attached to any one state can swing widely from episode to episode.

The TD target, $y = r + \hat{V}^\pi(s')$, uses the current, imperfect estimate $\hat{V}^\pi(s')$ in place of the true value of the next state. Since $\hat{V}^\pi$ can be wrong, especially early in training, this introduces bias: the target itself may be systematically off. But it only depends on one sampled reward, not an entire trajectory’s worth of randomness, so it has much lower variance.

• Monte Carlo: unbiased, high variance
• TD: some bias, lower variance

Monte Carlo trusts the data completely and pays for it in noise; TD trades some of that noise away by trusting its own, imperfect predictions instead.¹⁰

N-step returns: a middle ground

Monte Carlo and 1-step TD are two ends of a spectrum: sum every real reward all the way to $T$, or bootstrap after just one. Nothing forces that choice to be so extreme.

An n-step return sums the actual, observed rewards for the first $n$ steps, then bootstraps off $\hat{V}^\pi$ for everything after that:

$n=1$ recovers the 1-step TD target from above; $n=T$ recovers the Monte Carlo target, with no bootstrapping at all.

In practice, some $n$ strictly in between, $1 < n < T$, often works best: using $n$ real rewards before bootstrapping means less of the estimate leans on $\hat{V}^\pi$ being correct, so it has less variance than Monte Carlo; but it also depends on fewer of the single sampled trajectory’s rewards than the full return does, so it has lower bias than the 1-step bootstrap.

A full algorithm walkthrough

Putting every piece together needs one last bit of notation: the discount factor $\gamma \in [0, 1]$, which weights rewards further in the future slightly less than immediate ones (everywhere above has implicitly used $\gamma = 1$). With it, the actor-critic algorithm is:

Then repeat from step 1 with the newly updated policy.

Step 2’s target and loss are exactly the n-step return from above, now with discounting folded in:

Steps 1 and 4 are the actor: the policy network $\pi_\theta(a \mid s)$, updated by the policy gradient. Steps 2 and 3 are the critic: the value network $\hat{V}^\pi_\phi(s)$, whose bootstrapped estimate turns raw rewards into the advantage the actor learns from. Two separate networks, trained together, each taking the same state $s$ as input, giving the method its name.

Off-policy actor-critic

The algorithm above is on-policy: every gradient step needs a fresh batch of trajectories collected under the current policy $\pi_\theta$, including the batch the critic $\hat{V}_\phi^{\pi_\theta}$ is trained on. Turning this into an off-policy method means reusing data collected under an older policy for more than one gradient step, instead of collecting a batch, taking one gradient step, and throwing it away.¹¹ Each transition gets reused many times instead of once, which is far more sample-efficient. Importance sampling is what makes that reuse valid, at the cost of a slightly stale advantage estimate.

Version 1: Multiple Gradient Steps

This is what gives the version its name: instead of one gradient step per batch, steps 4 and 5 loop several times on $\theta'$ before the outer loop ever touches step 1 again. But the advantage $\hat{A}^{\pi_\theta}$ being reused in every one of those inner steps is still exactly what step 3 computed, from the old policy $\pi_\theta$; it never gets recomputed as $\theta'$ moves. The more inner steps you take, the more outdated that advantage becomes, and the less it reflects what $\theta'$ would actually experience now. $\theta'$ can’t wander too far from $\theta$ before this stops being reliable: once the two policies disagree enough, the importance weights also blow up, exactly the failure mode trust regions exist to prevent. In practice, “multiple” here means a handful of steps, not many.

After a handful of these inner steps, set $\theta \leftarrow \theta'$ and repeat from step 1 with a freshly collected batch.

Version 2: Replay Buffers

[Update this]

Proximal Policy Optimization

Recall the surrogate objective trick from policy gradients: instead of writing out a full gradient by hand, define a scalar objective whose gradient recovers it, so a single backward() call does the work. Applying that here, to the importance-weighted update from off-policy actor-critic, gives the surrogate objective

Differentiating $\tilde{J}$ with respect to $\theta'$, treating $\hat{A}^{\pi_\theta}$ as a constant, gives back exactly that importance-weighted policy gradient. As always, climbing $\tilde{J}$ increases the probability of high-advantage actions and decreases the probability of low-advantage ones.

$\tilde{J}$ has two pieces. The importance ratio,

compares the new policy to the old one. Unpacking each symbol:

• $s$ is the current state. For an LLM, that’s the prompt plus the tokens generated so far.
• $a$ is the action. For an LLM, usually the next token, or sometimes the entire generated sequence.
• $\pi_\theta$ is the old policy: the model as it was before the update.
• $\pi_{\theta'}$ is the new policy, the one being trained.
• $\pi(a \mid s)$ is the probability the policy assigns to taking action $a$ in state $s$.

So $r=1$ means $\theta'$ behaves exactly like $\theta$; $r>1$ means $\theta'$ is more likely to take this action than $\theta$ was; $r<1$ means less likely.¹² The advantage, $\hat{A}^{\pi_\theta}(s,a)$, was computed once from the old policy’s data: positive means the action was good, negative means it was bad. Multiplying them together says: increase the probability of good actions, decrease the probability of bad ones.

Why multiple gradient steps can go wrong

Version 1 already flagged the risk of repeating steps 4 and 5 too many times: nothing in $\tilde{J}$ itself penalizes $\theta'$ for drifting far from $\theta$, so an optimizer that’s free to keep climbing will happily push $r(\theta')$ far past $1$, long after the advantage estimate it’s multiplying has stopped being trustworthy.¹³

TRPO fixes this by directly constraining how far $\theta'$ is allowed to move, measured by KL divergence: $D_{\mathrm{KL}}(\pi_{\theta'} \,\|\, \pi_\theta) \leq \delta$. It works well, but solving a constrained optimization problem at every gradient step is mathematically involved.

PPO takes a simpler route: instead of constraining the policy directly, just stop the importance ratio itself from getting too large.

Clipping the importance ratio

Pick a small $\epsilon$ (commonly $\epsilon = 0.2$), and clip $r(\theta')$ to the interval $[1-\epsilon, 1+\epsilon]$ before it’s allowed to multiply the advantage: anything outside that interval gets pulled back to the nearest edge.¹⁴

Clipping matters because, left alone, $r(\theta') \, \hat{A}^{\pi_\theta}$ grows without bound: for a fixed positive advantage, a bigger ratio always means a bigger objective, so the optimizer is rewarded for pushing $\theta'$ arbitrarily far from $\theta$. Once the clipped ratio hits its boundary, it stops growing, so there’s no more reward for pushing it any further.¹⁵

The PPO objective

Taking the smaller of the clipped and unclipped versions, then summing back over the sampled timesteps the way $\tilde{J}$ did, gives the full PPO objective:

where $r_{t,i}(\theta')$ is the importance ratio at a single sampled state-action pair. The $\min$ and the clip both apply per sample, inside the sum: every term is capped on its own before anything is added up. The PPO paper writes this same objective as $L^{\mathrm{CLIP}}(\theta')$, with an expectation over samples in place of the sum.

Why take the minimum, instead of just always optimizing $\mathrm{clip}(r(\theta')) \, \hat{A}^{\pi_\theta}$ on its own? Because clipping alone can accidentally reward exactly the policy changes it’s supposed to discourage: whenever $r(\theta')$ has moved outside the trust region in a bad direction, the clipped term can score better than the true, unclipped one. Taking the minimum always keeps whichever number is worse, so a bad update can never be made to look better than it actually is.¹⁶

Either way, the policy can’t change too much in a single update while still improving the objective. This clipping mechanism, simple to implement and no constrained optimization required, is the core idea that makes PPO both stable and simple compared with earlier trust-region methods like TRPO.

What “the old policy” means in practice

The denominator of the ratio is not the model from the previous gradient step; it’s a frozen snapshot. Call it $P_0$. That frozen copy does two jobs: it generates the rollouts, and it supplies $\pi_\theta(a \mid s)$ for every ratio computed off them.

One pass of the loop looks like this:

That final reset is what keeps the batch honest: the actions and rewards in it came from $P_0$, so training on them long after the policy has moved to, say, $P_{20}$ means optimizing against behavior the current model no longer exhibits, and the gradient estimates get steadily less accurate.

The reason to squeeze several steps out of one batch at all is cost. For LLM RL, generating rollouts dominates: a single batch might mean generating 16,000 completions, scoring each one with a reward model or verifier, and computing advantages, which can take minutes. A gradient step on data already sitting in GPU memory is cheap by comparison. Taking exactly one update per batch would mean paying the expensive part over and over for one cheap step each time.

Despite the importance ratio, PPO is usually called an on-policy algorithm, because the data it trains on always comes from the current policy or one that’s only a few optimization steps old.