RTX Sensor Annotators#
The isaacsim.sensors.rtx extension uses Omniverse Replicator to provide Annotators for RTX Lidar and Radar data collection.
Annotators can be attached to render products, attached to OmniSensor prims (eg. OmniLidar or OmniRadar); for example,
when run in Script Editor, the following snippet creates an OmniLidar prim at /lidar, a render product for the sensor,
and attaches an IsaacExtractRTXSensorPointCloudNoAccumulator Annotator to the render product.
import omni
import omni.replicator.core as rep
from pxr import Gf
# Create an OmniLidar prim at prim path /lidar
_, sensor = omni.kit.commands.execute(
"IsaacSensorCreateRtxLidar",
translation=Gf.Vec3d(0.0, 0.0, 0.0),
orientation=Gf.Quatd(1.0, 0.0, 0.0, 0.0),
path="/lidar",
)
# Create a render product for the sensor.
render_product = rep.create.render_product(sensor.GetPath(), resolution=(1024, 1024))
# Create an annotator
annotator = rep.AnnotatorRegistry.get_annotator("IsaacExtractRTXSensorPointCloudNoAccumulator")
# Attach the render product after the annotator is initialized.
annotator.attach([render_product.path])
Annotators#
Each isaacsim.sensors.rtx Annotator is associated with a specific isaacsim.sensors.rtx OmniGraph node; refer
to that documentation for more information on the annotator’s outputs. This section details assumptions and limitations
for each annotator.
Note
In Isaac Sim 5.0, several existing isaacsim.sensors.rtx Annotators were removed in favor of simpler
Annotators that can handle output from the new OmniLidar or OmniRadar prims, in addition to the
deprecated Camera-prim-based workflows. See Deprecated Annotators for details.
Warning
If the Lidar rotation rate is slower than the frame rate, data from Annotators for accumulated Lidar scans will contain returns from multiple frames. If the Lidar prim moves between frames, or objects
move in the scene, the buffer may contain returns from before the Lidar or object(s) moved, causing points to appear as though they are “dragging” behind objects when
viewed with the DebugDrawPointCloud or DebugDrawPointCloudBuffer writers.
Warning
isaacsim.sensors.rtx Annotators rely on the GenericModelOutput AOV from the OmniLidar prim being
provided on device. If --/app/sensors/nv/lidar/outputBufferOnGPU or --/app/sensors/nv/radar/outputBufferOnGPU is
set to false, the Annotators will not function correctly.
IsaacCreateRTXLidarScanBuffer#
The IsaacCreateRTXLidarScanBuffer node accumulates frames of data from an OmniLidar prim into a single scan,
and provides the accumulated scan data as outputs. By default the node outputs a 3D Cartesian point cloud, and
can optionally the following data if the user sets the corresponding input flag on the node to true.
Note
The node outputs data as pointers to buffers, and the table below specifies the data type of each buffer.
Output |
Type |
Description |
Notes |
|---|---|---|---|
|
|
3D Cartesian point cloud. |
Always provided. |
|
|
Azimuth of each return, in degrees. |
Provided if |
|
|
Elevation of each return, in degrees. |
Provided if |
|
|
Range of each return, in world units (by default, meters). |
Provided if |
|
|
Intensity of each return, normalized as described here. |
Provided if |
|
|
Timestamp of each return, in nanoseconds since the start of the simulation. |
Provided if |
|
|
ID of the emitter that emitted the return. |
Provided if |
|
|
ID of the material of the object that generated the return. |
Provided if |
|
|
ID of the object that generated the return. |
Provided if |
|
|
Normal to the surface of the object that generated the return. |
Provided if |
|
|
Velocity of the object that generated the return. |
Provided if |
Note
Enabling nonzero normal output by setting --/app/sensors/nv/lidar/publishNormals=true will increase VRAM usage and may negatively impact performance.
Warning
In Isaac Sim 5.0, the normal and velocity outputs are not populated correctly by the IsaacCreateRTXLidarScanBuffer node. Users should instead use the GenericModelOutput AOV (see below). This issue will be fixed in a future release.
IsaacComputeRTXLidarFlatScan#
IsaacComputeRTXLidarFlatScan extracts depth and azimuth data from an accumulated 2D RTX Lidar scan.
Note
The IsaacComputeRTXLidarFlatScan Annotator does not support RTX Radar.
Warning
If the annotator is attached to a 3D Lidar (defined as having emitters at nonzero elevation angles), the annotator will not return any data.
Warning
Even if --/app/sensors/nv/lidar/outputBufferOnGPU=true is set, IsaacComputeRTXLidarFlatScanSimulationTime
output data will be on host memory.
IsaacExtractRTXSensorPointCloudNoAccumulator#
The IsaacExtractRTXSensorPointCloud node extracts the GenericModelOutput buffer’s point cloud data
into a Cartesian vector data buffer for each frame.
The IsaacExtractRTXSensorPointCloudNoAccumulator Annotator provides the node outputs via an Annotator.
Reading Data from the GenericModelOutput Buffer#
Note
Isaac Sim 4.5 included the OgnIsaacReadRTXLidarData node, which provided an
example of reading data from the GenericModelOutput buffer in Python. This node has been removed
as of Isaac Sim 5.0 and replaced by the utility module and functions described below.
The isaacsim.sensors.rtx.generic_model_output Python module provides APIs for inspecting the
GenericModelOutput buffer, generated by the GenericModelOutput annotator.
For more information on the GenericModelOutput buffer, see the API documentation..
For an example of reading data from the GenericModelOutput buffer from Isaac Sim, checkout the
standalone example located at standalone_examples/api/isaacsim.sensors.rtx/inspect_lidar_metadata.py.
Warning
The objId field returns a 128-bit stable unsigned integer corresponding to the prim path of the object
that generated the return. This field is provided as a numpy array of dtype np.uint8, and is only populated
if --/rtx-transient/stableIds/enabled=true is set. There are two known issues in Isaac Sim 5.0 regarding
objId output. First, the map from objId to prim path is not exposed to end-users, and the objId field is
populated with 1/16th of the necessary values. Second, the objId field is not populated for OmniLidar prims.
Both issues will be fixed in a future release.
Deprecated Annotators#
Several Annotators have been removed and/or replaced by the Annotators described above, as of Isaac Sim 5.0.
New Annotator outputs are not guaranteed to be the same as the outputs of the deprecated Annotators, gmoBufferPointer the table below describes affected Annotators and how to replace them.
Deprecated Isaac Sim 4.5 Annotator |
Replacement |
Details |
|---|---|---|
|
|
The new annotator outputs the same data as the old annotator. |
|
|
The new annotator outputs the same data as the old annotator. |
|
|
The new annotator outputs the same data as the old annotator, excluding |
|
|
See above. |
|
|
See above. |
|
|
See above. |
|
|
See Reading Data from the GenericModelOutput Buffer for details. |