Freeze when running benchmarks

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
,
1

I need assistance in fixing why my Jetson AGX Orin freezes midway whenever I run the yolov3tiny model using jetson-benchmarks. It becomes unresponsive and subsequently reboots after a few minutes. I have already eliminated the possibility of it being a power supply issue since I am using a power source that supplies more than the necessary power.I need assistance in understanding why my Jetson AGX Orin freezes midway whenever I run the yolov3tiny model using jetson-benchmarks. It becomes unresponsive and subsequently reboots after a few minutes. I have already eliminated the possibility of it being a power supply issue since I am using a power source that supplies more than the necessary power.I’ve try it for many times and every time it will freezed while running the benchmark

image
image1920×1440 182 KB

2

i have tried to jetson-gpu-burn for burning my jetson about half an hour but nothing wrong happend ,I am confused what’s wrong

4

Hi,

There are several configure in the GitHub.
Could you share which jetson-benchmarks command you run with us first?

Thanks.

5

sudo python3 /home/zyi/Desktop/jetson_benchmarks/benchmark.py --model_name tiny-yolov3 --csv_file_path /home/zyi/Desktop/jetson_benchmarks/benchmark_csv/orin-benchmarks.csv --model_dir /home/zyi/Desktop/jetson_benchmarks/models

6

I’m currently suspecting that the program’s invocation of dla0_core and dla1_core might be causing my Jetson to crash

7

I tested several times and found that if the system tries to utilize the cores dla0_core and dla1_core simultaneously, the system will freeze and reboot after several minutes. However, the system operates normally when deploying dla0_core or dla1_core separately.That’s a strange bug.What I should do to fix that.I wonder whether is the error of the driver or what

8

Hi,

Thanks for sharing the details.

We are trying to reproduce this internally.
Will update more info with you later.

9

Did you guys solve the problem about jetson orin agx freezing when the system deployinbg dlacore0 and dlacore1 simutaniously

10

Hi,

How do you check this issue is caused using DLA0 and DLA1 concurrently?
Do you launch it with trtexec rather than jetson_benchmarks?

Thanks.

11

Well, I test by deploying deeplearning-accelerator-sw, it pass all the exam that make me fell confused, I thought that the duration of launching dlacore were so short and it can’t express nothing because each time it only use one dlacore so can you run the benchmark and see whether the problem"

12

Hi,

The benchmark sample works correctly on AGX Orin.

$ sudo python3 benchmark.py --model_name tiny-yolov3 --csv_file_path /home/nvidia/topic_274094
Please close all other applications and Press Enter to continue...
Setting Jetson orin in max performance mode
gpu frequency is set from 306000000 Hz --> to 1300500000 Hz
dla frequency is set from 1600000000 Hz --> to 1536000000 Hz
sh: 1: cannot create /sys/devices/platform/pwm-fan: Is a directory
------------Executing yolov3-tiny-416------------

---------------------- 14.143577272727274 67.22893043478261 67.36969130434782
--------------------------

Model Name: yolov3-tiny-416 
FPS:3213.49 

--------------------------

Wall Time for running model (secs): 554.5775482654572

sh: 1: cannot create /sys/devices/platform/pwm-fan: Is a directory

Will share more info with you later.
Thanks.

13

Hi,

We found a possible issue.
Please try to set your device into the MAXN mode before running the benchmark.py script.

On the Orin device, rebooting is required when setting a different power mode.
However, there is an interactive checker to confirm the rebooting process which might make the script look like frozen.

To manually set the MAXN mode can avoid the issue.
Please give it a check to see if this works for your problem.

$ sudo nvpmodel -m 0
NVPM WARN: Golden image context is already created
NVPM WARN: Reboot required for changing to this power mode: 0
NVPM WARN: DO YOU WANT TO REBOOT NOW? enter YES/yes to confirm:

Thanks.

14

Sorry, I tried adjusting the settings, but the problem persists. I noticed that sometimes my Orin can pass the TinyYolo3 benchmark, but it occasionally freezes. Most of the time, my Jetson freezes when launching the test with ResNet at 1200x1200 resolution

15

Hi,

Could you share the reproducible step/source with us?
We have tried the tiny-yolov3 benchmark and it works without issue.

Thanks.

17

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.