← All posts

Scheduling Reinforcement Learning Training 1.73x Better

August 27, 2026 · Updated August 29, 2026

Note: This is a simulation. No real GPUs were used. GPU work is represented by asyncio.sleep().

The interesting part is the scheduling problem. I wanted to see how much time we could save by changing when each GPU does inference and backward work, without changing the training rules.

The setup

You don’t need to know much about GRPO for this. The simulator has:

• 12,800 rollouts, each producing one piece of training data.
• 4 GPUs, with 64 slots each.
• Inference, which takes 50–600 ms depending on the rollout.
• Backward, which takes about 230 ms.
• An optimizer step every 256 rollouts.

There is one constraint that drives the whole design:

A GPU can do inference or backward, but never both at once.

Backward takes all 64 GPU slots, so while it is running, that GPU is completely occupied.

The synchronous version

The baseline is straightforward. For each group of 256:

1. Run inference for all 256 rollouts.
2. Wait for all of them.
3. Run backward.
4. Wait again.
5. Take the optimizer step.

The problem is the waiting.

Inference latency varies from 50 to 600 ms. If one rollout takes 600 ms, the slots that finished in 50 ms sit idle until it does. Then the entire machine switches to backward, so no inference happens during that time.

There is also no overlap between batches: batch 2 cannot start until batch 1 is completely finished.

The asynchronous version

I changed the schedule rather than the work.

1. Split the GPUs. Two GPUs handle inference and two handle backward.
2. Refill slots immediately. When an inference finishes, that worker takes another rollout instead of waiting for the rest of the batch.
3. Use a pool. Finished inference goes into a ready queue. Backward workers pull chunks of up to 64 from it.
4. Optimize every 256 completed rollouts. They do not have to be the same 256 that started together.

The catch: don’t run too far ahead

There is a subtle failure mode. Rollouts remember which model version produced them. If inference gets too far ahead, it can generate almost everything with version 0.

Step 1: version-0 rollout → update to v1
Step 2: needs a v1 rollout → none available
Step 3: needs a v2 rollout → none available
...

That would be fast, but the model would only update once.

So the async loop has a few guardrails.

Admission cap. At most 512 rollouts are in flight with the default setup:

256 batch slots + (4 GPUs × 64 slots)
= 512

As the end of the rollout list approaches, the cap follows the amount of work remaining, with a minimum of batch_size + slots_per_gpu.

Freshest-first. Before an optimizer step, the code looks at the completed rollouts and makes sure the freshest available version is in the 256-rollout batch.

Rescue. If the optimizer is waiting for a newer version, the pipeline temporarily allows an extra inference and prioritizes the rollout that can unblock the optimizer. Without this, the pipeline can deadlock: the optimizer waits for fresh data, while fresh data cannot be generated because the admission limit is full.

Results

Same simulated hardware, same 12,800 rollouts, same optimizer requirements.

1.73x faster. All 12,800 rollouts were processed and all 50 optimizer steps happened.

The absolute minutes are simulator numbers, so I would not read them as a prediction for real hardware. The useful measurement is the 1.73x reduction in simulated wall-clock time.

What I learned

Look for barriers. The biggest win came from removing “wait for everything” points, not from complicated code.

Separate requirements from implementation. The rule was “256 rollouts per optimizer step.” It did not say which 256. That gave us room to build a pool instead of fixed batches.

Measure the schedule. I also tried a replay-buffer design. It looked similar on paper but was slower: 1.44x instead of 1.78x in that run, because it tied rollouts to groups and reintroduced stragglers.

Test strange configurations. I tried 1 GPU, 7 GPUs, 1 slot per GPU, very large groups, tiny groups, and repeated runs. That exposed an edge case where a backward pass larger than a whole group can cause several early model updates to be skipped.