Population-Based Training

Population-based training (PBT) is an online optimization loop around regular reinforcement learning (RL) training. Instead of choosing one set of hyperparameters and training one policy, PBT trains a population of independent policies on the same task. It periodically ranks them by a task-level objective. An underperforming learner copies a promising learner’s checkpoint, mutates selected hyperparameters, and continues training from the copied weights.

This cycle combines two ideas:

• Explore: different seeds and hyperparameters let the learners discover different behaviors.

• Exploit: checkpoint replacement redirects compute from stalled learners to behaviors that are already working.

Unlike a fixed random or grid search, PBT can change hyperparameters during training and reuse the weights from a successful run. The final artifact is still a normal policy checkpoint; PBT changes how it is trained, not how it is deployed.

See also

If you are new to RL workflows in Isaac Lab, first read Training with an RL Agent and Configuring an RL Agent.

Caution

Isaac Lab currently supports PBT only with the RL-Games library.

When to Use PBT

PBT is most useful when:

• independent seeds frequently converge to very different results or stall in local optima;

• a task has a stable scalar success metric that can rank policies while they train;

• useful hyperparameters may need different values in early and late training; and

• enough compute is available to train several policies concurrently.

Start with ordinary single-policy training when a known configuration converges reliably, when compute only supports one or two runs, or when policies cannot be compared with a meaningful objective. PBT adds checkpoint I/O and coordination overhead, and a population of N learners consumes approximately the aggregate compute of N ordinary runs. A fair experiment therefore compares PBT with the same number of independent fixed-configuration runs, not with one run.

PBT is also different from multi-GPU training. Distributed data-parallel training uses several devices to update one policy. PBT gives each learner its own policy, optimizer state, experience, and checkpoint, and only exchanges checkpoints during selection.

Example Where PBT Was Enabling

The DexPBT study tested contact-rich manipulation tasks with large observation and action spaces, sparse task success, and many locally useful behaviors that did not lead to a complete solution. For its PBT-versus-PPO comparison, the authors held aggregate compute constant: the best learner in an eight-policy PBT population was compared with the best of eight independent PPO seeds, with every learner trained for five billion environment steps.

Single-arm object reorientation comparison reported in Figure 4 of the DexPBT study

Condition	Training behavior	Outcome under the study budget
PPO without PBT	Eight policies trained independently with fixed hyperparameters; discoveries were not shared.	None of the eight runs reached the final 1 cm success tolerance.
PPO with PBT	Eight learners periodically shared promising weights and explored mutated hyperparameters.	The population reached the final tolerance and continued improving the number of consecutive successes.

The study reported improvement in every tested scenario and found PBT to be an enabling factor for non-trivial performance in three of them. In the harder dual-arm reorientation task, its best PBT policy reached almost 40 of 50 consecutive successes. See the paper’s training curves and supplementary policy videos for the full comparison and learned behaviors.

No task mathematically requires PBT. Here, “enabling” means that fixed PPO did not solve the task within the tested budget while PBT did. These experiments used the Isaac Gym DexPBT tasks; they motivate the Isaac Lab workflow below but are not benchmark results for the included Shadow Hand example.

How Isaac Lab Selects Policies

Every interval_steps, each learner saves its checkpoint and current objective to a shared workspace. For the initialized policies, let the objective mean and standard deviation be \(\mu\) and \(\sigma\). Isaac Lab computes the leader and underperformer cuts as:

\[U = \max(\mu + \mathtt{threshold\_std}\,\sigma,\ \mu + \mathtt{threshold\_abs})\]

\[L = \min(\mu - \mathtt{threshold\_std}\,\sigma,\ \mu - \mathtt{threshold\_abs})\]

Policies with an objective greater than \(U\) are leaders; policies below \(L\) are underperformers. Only an underperformer is changed. It copies a randomly selected leader and mutates each whitelisted hyperparameter with probability mutation_rate. If there is no leader, it keeps its own checkpoint and mutates its hyperparameters. All other policies continue unchanged.

The learners coordinate through checkpoint and YAML metadata files rather than a central process. They may progress asynchronously: a learner compares checkpoints from the latest PBT iteration available at or before its own. Every learner must still use the same num_policies and must see the shared directory at the same absolute path because checkpoint paths stored in the metadata are absolute.

Choose the Ranking Objective

The objective controls selection pressure, so choose a metric that measures the behavior you ultimately want and is comparable across the population. Higher values are always considered better. A task-success metric is usually more suitable than a dense shaped reward whose scale changes during training.

The included reorientation configuration uses:

objective: episode.consecutive_successes

This dotted path resolves to infos["episode"]["consecutive_successes"]. The RL-Games wrapper remaps extras["log"] to infos["episode"], so use the episode. prefix even when the environment writes the metric to extras["log"].

Before a long run, verify that the objective is present, scalar, and increases when policy behavior improves. A noisy or non-stationary objective can repeatedly replace useful policies. PBT population members also stop being independent after checkpoint exchange, so evaluate the selected checkpoint separately across held-out seeds and conditions.

Configure PBT

The Isaac-Reorient-Cube-Shadow-Direct RL-Games configuration provides a complete PBT example. PBT is disabled by default; the launch command in the next section enables it.

pbt:
  enabled: False
  policy_idx: 0
  num_policies: 8
  directory: .
  workspace: "pbt_workspace"
  objective: "episode.consecutive_successes"
  interval_steps: 50000000
  threshold_std: 0.1
  threshold_abs: 0.025
  mutation_rate: 0.25
  change_range: [1.1, 2.0]
  mutation:
    agent.params.config.learning_rate: "mutate_float"
    agent.params.config.grad_norm: "mutate_float"
    agent.params.config.entropy_coef: "mutate_float"
    agent.params.config.critic_coef: "mutate_float"
    agent.params.config.bounds_loss_coef: "mutate_float"
    agent.params.config.kl_threshold: "mutate_float"
    agent.params.config.gamma: "mutate_discount"
    agent.params.config.tau: "mutate_discount"

The main controls are:

directory and workspace

Identify the shared storage root and the PBT run within that root. Use a new workspace for each population.

num_policies and policy_idx

Define the population size and this learner’s unique zero-based index.

objective

Selects the scalar from the environment info dictionary used to rank learners.

interval_steps

Sets the number of environment steps between checkpoint comparisons. Shorter intervals react sooner but cause more checkpoint I/O and give a mutated learner less time to adapt.

threshold_std and threshold_abs

Control how far a policy must be from the population mean to be classified as a leader or underperformer. If no policy crosses both effective margins, the population continues without replacement.

mutation_rate, change_range, and mutation

Control which hyperparameters may change and how often. mutate_float multiplies or divides by a random factor in change_range; mutate_discount conservatively changes the distance from 1.0 for discount-like values. Parameters absent from mutation remain fixed.

Launch the Population

Start one process per policy. The example below launches policy 0 of the default eight-policy population. Run it in eight terminals or scheduler jobs, changing only PBT_POLICY_IDX from 0 through 7. Keep PBT_NUM_POLICIES, PBT_DIRECTORY, and agent.pbt.workspace identical for every process.

export PBT_POLICY_IDX=0
export PBT_NUM_POLICIES=8
export PBT_DIRECTORY=/absolute/path/to/shared_folder

uv run --extra rl-games isaaclab train --rl_library rl_games \
  --task Isaac-Reorient-Cube-Shadow-Direct \
  --num_envs 8192 \
  --seed="${PBT_POLICY_IDX}" \
  agent.pbt.enabled=true \
  agent.pbt.num_policies="${PBT_NUM_POLICIES}" \
  agent.pbt.policy_idx="${PBT_POLICY_IDX}" \
  agent.pbt.directory="${PBT_DIRECTORY}" \
  agent.pbt.workspace=shadow_reorient_pbt

Assign each process a GPU with your scheduler or CUDA_VISIBLE_DEVICES when running several workers on one host. Weights & Biases tracking is optional; if enabled, give each worker a unique run name while keeping the same project.

During training, each policy writes numbered .pth checkpoints and matching .yaml metadata under its policy directory in the shared workspace. The metadata records the objective and mutated parameter values. At the end of the run, select the checkpoint with the best validated objective, then evaluate it as a normal RL-Games policy.

Troubleshooting

• No policies are replaced: the population may not yet have crossed the configured margins. Inspect the objectives before reducing the thresholds; identical low scores can also mean the ranking metric is uninformative.

• An objective lookup fails: verify the dotted path against the keys emitted in infos["episode"].

• Workers cannot load one another’s checkpoints: confirm that every worker uses the same absolute shared path, workspace name, and population size.

• Training spends too much time checkpointing: increase interval_steps or place the workspace on faster shared storage.

References

• Jaderberg et al., Population Based Training of Neural Networks (2017).

• Petrenko et al., DexPBT: Scaling up Dexterous Manipulation for Hand-Arm Systems with Population Based Training (2023).