Mastering Omniverse for Robotics#
NVIDIA Omniverse offers a large suite of tools for 3D content workflows. There are three main components (relevant to robotics) in Omniverse:
USD Composer: This is based on a novel file format (Universal Scene Description) from the animation (originally Pixar) community that is used in Omniverse
PhysX SDK: This is the main physics engine behind Omniverse that leverages GPU-based parallelization for massive scenes
RTX-enabled Renderer: This uses ray-tracing kernels in NVIDIA RTX GPUs for real-time physically-based rendering
Of these, the first two require a deeper understanding to start working with Omniverse and its constituent applications (Isaac Sim and Isaac Lab).
The main things to learn:
How to use the Composer GUI efficiently?
What are USD prims and schemas?
How do you compose a USD scene?
What is the difference between references and payloads in USD?
What is meant by scene-graph instancing?
How to apply PhysX schemas on prims? What all schemas are possible?
How to write basic operations in USD for creating prims and modifying their attributes?
Part 1: Using USD Composer#
While several video tutorials and documentation exist out there on NVIDIA Omniverse, going through all of them would take an extensive amount of time and effort. Thus, we have curated these resources to guide you through using Omniverse, specifically for robotics.
Introduction to Omniverse and USD
Using Omniverse USD Composer
Materials and MDL
Omniverse Physics and PhysX SDK
Importing assets
Part 2: Scripting in Omniverse#
The above links mainly introduced how to use the USD Composer and its functionalities through UI operations. However, often developers need to write scripts to perform operations. This is especially true when you want to automate certain tasks or create custom applications that use Omniverse as a backend. This section will introduce you to scripting in Omniverse.
USD is the main file format Omniverse operates with. So naturally, the APIs (from OpenUSD) for modifying USD are at the core of Omniverse. Most of the APIs are in C++ and Python bindings are provided for them. Thus, to script in Omniverse, you need to understand the USD APIs.
Note
While Isaac Sim and Isaac Lab try to “relieve” users from understanding the core USD concepts and APIs, understanding these basics still help a lot once you start diving inside the codebase and modifying it for your own application.
Before diving into USD scripting, it is good to get acquainted with the terminologies used in USD. We recommend the following introduction to USD basics by Houdini, which is a 3D animation software. Make sure to go through the following sections:
As a test of understanding, make sure you can answer the following:
What are prims? What is meant by a prim path in a stage?
How are attributes related to prims?
How are schemas related to prims?
What is the difference between attributes and schemas?
What is asset instancing?