Incorrect BBoxes in BasicWriter output

Turowicz · November 4, 2024, 11:35am

I’m having a very difficult situation where the data that comes out of the BasicWriter is incorrect in term of BBox sizes and instance segmentation. I don’t even know how to start digging into this as the data in the Isaac application seems all correct when I debug. It only fails when I run a docker container as a job.

The problem seems to be only with class=person. All other aspects of the asset are ok.

Lets start with the problem:

EMT in the red t-shirt has a class=person BBox only for the head and shirt, but not for the legs and arms:

image1672×1172 120 KB
Guy in vest only has a class=person BBox for the chest and arms and head but not the legs. The issue also shows on instance segmentation:
image1665×1123 161 KB

image149×158 2.79 KB

The persons are instantiated on the scene with:

with rep.create.from_usd(usd=f"file://{self.nucleus}/Library/srv_persons.usd",count=config.personCount):
    rep.modify.pose(rotation=rep.distribution.uniform([0, 0, -180],[0, 0, 180]))
    rep.modify.variant(name="person",value=rep.distribution.choice(choices=self.person_variants))
    rep.randomizer.scatter_2d(seed=seed,surface_prims=floor(),check_for_collisions=True,min_samp=config.scatterRange[0],max_samp=config.scatterRange[1])

The asset itself contains hardcoded class=person semantic on the agg_persons, so its not possible for it to “disappear” or get modified.

As you see the srv_person asset contains reference that is controlled by variant set. Depending on a variant selected we get a different person model loaded up. The semantic class=person is on the agg_persons prim, therefore it should always be included in the BasicWriter output.

This issue seems to impact about 1%-2% of output annotations.

I can’t wrap my head around it.

Please help.

Turowicz · November 5, 2024, 8:04am

Actually its not only class=person. We are also missing class=helmet and others:

pcallender · November 8, 2024, 5:48pm

Talking with the replicator engineers about this … Have you ever seen this issue when variants are not used? Are you able to pass me the usda? We’d like to take a look at the prim structure/hierarchy. Where are you applying the semantics and where are the variants residing?

Turowicz · November 12, 2024, 11:27am

I’ve run the following experiments:

people: OK
people with accessories: OK
people with accessories + clutter: BUG
people with accessories + pallet jacks: OK
people with accessories + pallet jack riding: OK
people with accessories + traffic lights: OK

Seems like the “+ clutter” one is causing trouble. The cluter is as follows:

from base_spawner import BaseSpawner
from scenes.base_scene_config import SceneConfig
from omni.replicator.core.scripts.utils.utils import ReplicatorItem

import omni.replicator.core as rep

CLUTTER = [
    # "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Demos/AEC/BrownstoneDemo/Props/Hydrant/Hydrant.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Machinima/PostScriptum/Maps/EntryMap/Props/SM_trashcans.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/NVIDIA/Assets/ArchVis/Residential/Outdoors/Planters/SquareGardenPlanter_Tall.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/NVIDIA/Assets/ArchVis/Residential/Outdoors/Planters/Anza_Large.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/NVIDIA/Assets/ArchVis/Residential/Outdoors/Planters/ChippedPlanter_Large.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Machinima/PostScriptum/Maps/EntryMap/Props/SM_Workshop_Ladder.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Machinima/PostScriptum/Maps/EntryMap/Props/WoodenBox_Stack_C.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Machinima/BannerlordII/Props/market_potteryPile4.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Machinima/BannerlordII/Props/weapon_rack_vlandia_a.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Machinima/Squad/Stages/Desert/Props/Ace_Van.usd",
    # "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/NVIDIA/Assets/Vegetation/Trees/Common_Apple.usd",
    # "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/NVIDIA/Assets/Vegetation/Trees/Red_Oak.usd",
    # "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/NVIDIA/Assets/Vegetation/Trees/Japanese_Maple_Fall.usd",
]

CLUTTER_SM = [
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Simple_Warehouse/Props/SM_BarelPlastic_B_03.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Simple_Warehouse/Props/SM_BarelPlastic_C_02.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Simple_Warehouse/Props/SM_BottlePlasticA_02.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Simple_Warehouse/Props/SM_BottlePlasticB_01.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Simple_Warehouse/Props/SM_BottlePlasticA_02.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Simple_Warehouse/Props/SM_BottlePlasticA_02.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Simple_Warehouse/Props/SM_BottlePlasticD_01.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Simple_Warehouse/Props/SM_BottlePlasticE_01.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Simple_Warehouse/Props/SM_BucketPlastic_B.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Props/Forklift/forklift.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Hospital/Props/Pharmacy_Low.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Hospital/Props/SM_BedSideTable_01b.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Hospital/Props/SM_BooksSet_26.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Hospital/Props/SM_BottleB.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Hospital/Props/SM_BottleA.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Hospital/Props/SM_BottleC.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Hospital/Props/SM_Cart_01a.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Hospital/Props/SM_Chair_02a.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Hospital/Props/SM_Chair_01a.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Hospital/Props/SM_Computer_02b.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Hospital/Props/SM_Desk_04a.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Hospital/Props/SM_DisposalStand_02.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Hospital/Props/SM_FirstAidKit_01a.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Hospital/Props/SM_GasCart_01c.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Hospital/Props/SM_Gurney_01b.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Hospital/Props/SM_HospitalBed_01b.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Hospital/Props/SM_MedicalBag_01a.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Hospital/Props/SM_Mirror.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Hospital/Props/SM_MopSet_01b.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Hospital/Props/SM_SideTable_02a.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Hospital/Props/SM_SupplyCabinet_01c.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Hospital/Props/SM_SupplyCart_01e.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Hospital/Props/SM_TrashCan.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Hospital/Props/SM_Washbasin.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Hospital/Props/SM_WheelChair_01a.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Office/Props/SM_WaterCooler.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Office/Props/SM_TV.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Office/Props/SM_TableC.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Office/Props/SM_Recliner.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Office/Props/SM_Personenleitsystem_Red1m.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Office/Props/SM_Lamp02_162.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Office/Props/SM_Lamp02.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Office/Props/SM_HandDryer.usd",
    "http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Office/Props/SM_Extinguisher.usd",
]

class ClutterSpawner(BaseSpawner):
    def on_trigger_inner(self, seed : int, floor : lambda : ReplicatorItem, config : SceneConfig, clutter : list[str], scale_lower, scale_upper):
        seed = seed + 1
        with rep.randomizer.instantiate(paths=clutter,size=int(config.clutterCount / 2),mode="scene_instance"):
            rep.modify.pose(scale=rep.distribution.uniform(scale_lower,scale_upper),
                            rotation=rep.distribution.uniform([0, 0, -180],[0, 0, 180]))
            rep.randomizer.scatter_2d(seed=seed,surface_prims=[floor()],check_for_collisions=True,min_samp=config.scatterRange[0],max_samp=config.scatterRange[1])
        return seed

    def on_trigger(self, seed : int, floor : lambda : ReplicatorItem, config):
        if config.clutterCount > 1:
            seed = self.on_trigger_inner(seed, floor, config, CLUTTER, 0.8, 1.2)
            seed = self.on_trigger_inner(seed, floor, config, CLUTTER_SM, 80, 120)

As you can see we rely only on the NVIDIA-provided content for clutter.

I am now running more experiments to make sure this is the right track:

people with accessories + pallet jacks + pallet jack riding+ traffic lights

@pcallender I am sending you the usd on private message. Copywright protected content.

Its VERY STRANGE as it seems to break PEOPLE-related annotations when CLUTTER is spawned.

What I can say the structure of a person asset has been modified by our artists and perhaps its in conflict with something from clutter?

Turowicz · November 12, 2024, 2:56pm

That worked fine. It looks like the clutter is breaking the bboxes for people and their accessories.

Turowicz · November 14, 2024, 11:17am

It looks like decreasing clutter count from 100 to 10 “fixes” the problem. Please tell me I am dreaming - this seems like a massive flaw in the tool.

Turowicz · November 15, 2024, 2:46pm

Can it be that its just too many objects on the scene that breaks the BasicWriter?

Turowicz · November 18, 2024, 10:38am

Or asset tree depth?

Turowicz · November 18, 2024, 10:39am

@pcallender @jlafleche @hclever @dennis.lynch ?

Turowicz · November 18, 2024, 10:39am

@pcallender I sent you the usds on private.

Turowicz · November 25, 2024, 9:47am

Can I please get any help with this? We have invested so much into Omniverse that we are past the point of no return. This situation is blocking us from actually using the tool. This is a terrible situation and we need your help to progress. We are happy to pay for your engineer’s time to come work on this for a couple of weeks. @pcallender @jlafleche @hclever @dennis.lynch

Turowicz · December 2, 2024, 9:51am

Heeeeelp

pcallender · December 2, 2024, 5:12pm

I’m roping some others in to see if we can pin this down and find out what’s going on.

pcallender · December 4, 2024, 3:34pm

Quick update. We think this might be the same or similar to an issue we’ve been trying to track down also, with a very low repro rate. The engineers think there’s something odd going on in the acceleration structure.

I have the USDs you DM’d me, but I need a bit more assistance:

Can you DM or attach the full script you’re using, or a minimal script that you’ve verified demonstrates the issue. I realize you’ve got the two code snippets above, but I don’t want to leave anything to chance.
Link me the exact docker file you’re using.
Drop/DM me a log if you’re able, ideally one that was active when the missing bbox happens in a dataset generation.

Basically we need to get as close to your repro as possible without doing extra stitching of code snippets or relinking assets, and ultimately we need to repro this locally to really debug this.

Thanks in advance, lets see if we can get this solved.

Turowicz · December 4, 2024, 3:41pm

Would you be open to doing a Teams (or anything else) session to demonstrate the issue? I can’t share the proprietary codebase and writing a script to reproduce could prove to be impossible.

I have DMed you a link to my calendar.

Turowicz · December 11, 2024, 10:14am

@jlafleche & @pcallender thank you for yesterday’s chat. I will be now experimenting with asset pooling in order to eliminate the strain on the USD loader. One thing that comes to mind though is that my “container asset” srv_persons.usd contains variants with references to assets. And those people assets contain variants with references to accessories. Does this mean the loader will still have to do “problematic” loading on each permutation and probably pooling that “container asset” will not be enough? Should I flatten this structure somehow and use payloads instead of references to make it all load on start?

jlafleche · December 12, 2024, 5:55pm

My suggestion is to create K assets (say 10,000) with randomly assigned variants. As long as the assets are created with the assigned variant before the first frame is rendered, the renderer will get those meshes on creation and won’t need to deal with meshes being removed/added from the simulation. From the renderer’s perspective, it shouldn’t make a difference whether the asset is structured with variants, payloads or references.

Turowicz · December 13, 2024, 8:30am

Thanks I will now try that. Could you tell me if there will ever be a fix for this? I am wondering about restructuring my application completely and this is an important aspect

jlafleche · December 19, 2024, 2:51pm

Yes, we’re absolutely working towards a fix - it’s imperitive the groundtruth being reported is correct. That said, the above approach has the advantage that by avoiding creation/deletion of USD primitives, you can gain a meaningful speedup in generation.

Turowicz · January 7, 2025, 11:20am

@jlafleche I’m having trouble getting this to work. Here is my attempt but the assets aren’t being scattered on the surface - its empty:

from base_spawner import BaseSpawner
from scenes.base_scene_config import SceneConfig
from entities.person import Person, get_persons_variants
from omni.replicator.core.scripts.utils.utils import ReplicatorItem

import omni.client as client
import omni.replicator.core as rep

TOTAL=10
MINUS_MILION=-1000000.0

class PersonSpawner(BaseSpawner):
    assets = list[Person]

    def __init__(self, nucleus : str):
        super().__init__()
        self.nucleus = nucleus
        self.asset_url = f"{nucleus}/Library/_humans_agg"
        self.animation_url = f"{nucleus}/Library/People"
        self.semantic_url = f"{nucleus}/Library/Semantics"
        self.person_variants : list[str] = []
        self.pool : list[ReplicatorItem] = []
        self.poolGroup : ReplicatorItem

    async def load(self, config : SceneConfig):
        if config.personCount > 0:
            for person in await get_persons_variants(self.nucleus):
                if f"Motions/person" in person.animation:
                    self.person_variants.append(person.variant)

            for i in range(TOTAL):
                self.pool.append(rep.create.from_usd(usd=f"file://{self.nucleus}/Library/srv_persons.usd"))
                
                with self.pool[i]:
                    rep.modify.pose(position=[MINUS_MILION + i, MINUS_MILION + i, MINUS_MILION + i])

            self.poolGroup = rep.create.group(items=self.pool)

            with self.poolGroup:
                rep.modify.variant(name="person",
                                   value=rep.distribution.choice(choices=self.person_variants))

    def on_trigger(self, seed : int, floor : lambda : ReplicatorItem, config : SceneConfig):
        if config.personCount > 0:
            with rep.distribution.choice(choices=self.pool,num_samples=config.personCount):
                rep.modify.pose(rotation=rep.distribution.uniform([0, 0, -180],[0, 0, 180]))
                rep.randomizer.scatter_2d(seed=seed,
                                          surface_prims=floor(),
                                          check_for_collisions=True,
                                          min_samp=config.scatterRange[0],
                                          max_samp=config.scatterRange[1])

Topic		Replies	Views
SUPERBUG: IsaacSim container 4.0.0 is randomly skipping semantic data (python) Synthetic Data Generation (SDG) python , omniverse	9	85	September 5, 2024
ValueError: Unable to set target prim of type <class 'pxr.Usd.Prim'> in Isaac Sim Script Editor Isaac Sim	12	1096	November 1, 2023
Omni.replicator.core.scripts.randomizer has no attribute scatter_boxes Isaac Sim	2	1166	January 31, 2023
Enabling omni.anim.skelJoint crashes the app Synthetic Data Generation (SDG)	55	115	August 27, 2024
Scatter_2d doesn't have a scale parameter for objects? Isaac Sim	5	574	March 28, 2023
Isaac v4.0.0 Container cannot load our YAML scenarios from v2023.1.1 Synthetic Data Generation (SDG)	11	300	August 8, 2024
Scatter3d on scatter2d-ed volume meshes Synthetic Data Generation (SDG)	6	35	March 3, 2025
Objects Sinking into one another (replicator python API) Isaac Sim python , isaac-sim-v4-2-0	4	55	May 28, 2025
Using scatter_2d in Isaac Sim Script Editor Isaac Sim	1	467	September 11, 2023
Retrieving replicator value Isaac Sim	13	854	April 17, 2024

Incorrect BBoxes in BasicWriter output

Related topics