Direct and Manager MDP

Direct and Manager MDP#

A reinforcement learning task defines a Markov decision process (MDP): the actions an agent can take, the observations it receives, its rewards, and when an episode ends. In Isaac Lab, you can write this logic directly in an environment class or assemble it from terms coordinated by managers.

Both approaches use Isaac Lab’s simulation, scene, assets, and sensors. The choice is how you organize the task code; it does not select a physics backend.

Choosing an approach#

Direct

Manager-based

Task logic

Implement methods on an environment class.

Configure reusable terms; managers run them.

Changing a task

Edit the methods and helpers that implement its behavior.

Add, replace, or tune terms in its configuration.

A good fit

Closely coupled logic, custom optimization, or multi-agent tasks.

Tasks that share observations, rewards, actions, or reset behavior.

For a new single-agent task, manager-based environments are a useful starting point: you can reuse existing terms and add your own where needed. Choose direct when you prefer to keep the task logic together or need more control over its implementation. Performance depends on the work your task does, so profile before choosing a workflow for speed alone.

Direct environments#

Subclass envs.DirectRLEnv and implement the methods that define the task. The base class handles the simulation loop and calls your methods at the appropriate point in each step.

Direct: one environment class defines actions, observations, rewards, episode endings, and resets.

  • _pre_physics_step() processes incoming actions; _apply_action() applies them.

  • _get_observations() and _get_rewards() return observations and rewards.

  • _get_dones() reports terminations and timeouts; _reset_idx() resets selected environments.

Use envs.DirectRLEnvCfg for the environment configuration. For multi-agent tasks, use envs.DirectMARLEnv and envs.DirectMARLEnvCfg.

The methods can call shared helpers or optimized PyTorch or Warp code. Keeping the task in one class does not mean every calculation must be written inline. This structure is also familiar when porting tasks from IsaacGymEnvs or OmniIsaacGymEnvs.

Cartpole rewards: direct

Cartpole computes its reward in _get_rewards().

def _get_rewards(self) -> torch.Tensor:
    return compute_rewards(
        self.cfg.rew_scale_alive,
        self.cfg.rew_scale_terminated,
        self.cfg.rew_scale_pole_pos,
        self.cfg.rew_scale_cart_vel,
        self.cfg.rew_scale_pole_vel,
        self.joint_pos[:, self._pole_dof_idx],
        self.joint_vel[:, self._pole_dof_idx],
        self.joint_vel[:, self._cart_dof_idx],
        self.reset_terminated,
        self.step_dt,
    )

The calculation lives in a separate, TorchScript-compiled compute_rewards() function.

@torch.jit.script
def compute_rewards(
    rew_scale_alive: float,
    rew_scale_terminated: float,
    rew_scale_pole_pos: float,
    rew_scale_cart_vel: float,
    rew_scale_pole_vel: float,
    pole_pos: torch.Tensor,
    pole_vel: torch.Tensor,
    cart_vel: torch.Tensor,
    reset_terminated: torch.Tensor,
    step_dt: float,
) -> torch.Tensor:
    pole_pos = wrap_to_pi(pole_pos)
    rew_alive = rew_scale_alive * (1.0 - reset_terminated.float())
    rew_termination = rew_scale_terminated * reset_terminated.float()
    rew_pole_pos = rew_scale_pole_pos * torch.sum(torch.square(pole_pos), dim=-1)
    rew_cart_vel = rew_scale_cart_vel * torch.sum(torch.abs(cart_vel), dim=-1)
    rew_pole_vel = rew_scale_pole_vel * torch.sum(torch.abs(pole_vel), dim=-1)
    total_reward = (rew_alive + rew_termination + rew_pole_pos + rew_cart_vel + rew_pole_vel) * step_dt
    return total_reward

Manager-based environments#

Use envs.ManagerBasedRLEnv with a envs.ManagerBasedRLEnvCfg to assemble a task from configured terms. A term defines a piece of behavior, such as an observation or reward. Its configuration selects the implementation and supplies parameters; the corresponding manager coordinates execution and combines the results.

Manager-based: task configuration selects reusable terms coordinated by action, observation, and reward managers.

For example, a reward manager evaluates reward terms and combines their weighted contributions. An observation manager collects observation terms into groups. Other managers handle actions, terminations, events such as resets, commands, and curricula.

To change one part of the task, replace a term or adjust its parameters. The same term can serve several tasks, and custom terms can contain whatever calculation the task needs.

Cartpole rewards: manager-based

Cartpole combines reward terms in RewardsCfg. Each term specifies a function, a weight, and any parameters the function needs.

@configclass
class RewardsCfg:
    """Reward terms for the MDP."""

    # (1) Constant running reward
    alive = RewTerm(func=mdp.is_alive, weight=1.0)
    # (2) Failure penalty
    terminating = RewTerm(func=mdp.is_terminated, weight=-2.0)
    # (3) Primary task: keep pole upright
    pole_pos = RewTerm(
        func=mdp.joint_pos_target_l2,
        weight=-1.0,
        params={"asset_cfg": SceneEntityCfg("robot", joint_names=["cart_to_pole"]), "target": 0.0},
    )
    # (4) Shaping tasks: lower cart velocity
    cart_vel = RewTerm(
        func=mdp.joint_vel_l1,
        weight=-0.01,
        params={"asset_cfg": SceneEntityCfg("robot", joint_names=["slider_to_cart"])},
    )
    # (5) Shaping tasks: lower pole angular velocity
    pole_vel = RewTerm(
        func=mdp.joint_vel_l1,
        weight=-0.005,
        params={"asset_cfg": SceneEntityCfg("robot", joint_names=["cart_to_pole"])},
    )
    # (6) Success rate tracking (zero-weight, metric only)
    success_rate = RewTerm(func=mdp.survival_success_rate, weight=0.0)

For environments that do not need RL rewards and terminations, use envs.ManagerBasedEnv with envs.ManagerBasedEnvCfg.

Where to go next#