I am using YOLOv8 inference on Jetson Orin NX 16GB running Python. The GPU however does not take the load and instead the CPU is still utilized.
I imagine CUDA compatibility is an issue here.
Jetpack 6.0, but I manually installed CUDA12.2, cudnn9.0, PyTorch 2.2.1, Torchvision 0.17, CudaCompat-12.2
As per the instruction provided by NVIDIA and PyTorch.
wget https://developer.download.nvidia.com/compute/cudnn/9.0.0/local_installers/cudnn-local-tegra-repo-ubuntu2204-9.0.0_1.0-1_arm64.debsudo dpkg -i cudnn-local-tegra-repo-ubuntu2204-9.0.0_1.0-1_arm64.debsudo cp /var/cudnn-local-tegra-repo-ubuntu2204-9.0.0/cudnn-*-keyring.gpg /usr/share/keyrings/sudo apt-get updatesudo apt-get -y install cudnn
sudo apt-get -y install cudnn-cuda-12
The initilization in the script reads:
import cv2
import pandas as pd
import torch
from ultralytics import YOLO
from tracker import*
import paho.mqtt.client as mqtt
import json
import cudatracker = Tracker() # tracks objects and assigns IDs to them
model = YOLO(‘trial_29_yolov8_s_best.pt’)
I have read about specifically setting a target device for cuda but no torch.cuda device was detected.
torch.cuda.set_device(0)
Sample scripts I use to confirm the utilization of CUDA lead to negative results.
import torch
def print_gpu_properties():
if torch.cuda.is_available():
num_devices = torch.cuda.device_count()
print(f"Number of CUDA devices: {num_devices}“)
for i in range(num_devices):
device = torch.device(f"cuda:{i}”)
properties = torch.cuda.get_device_properties(device)
print(f"Device {i} properties:“)
print(f”\tName: {properties.name}“)
print(f”\tTotal memory: {properties.total_memory / (1024 ** 3):.2f} GB")
print(f"\tCompute capability: {properties.major}.{properties.minor}")
else:
print(“CUDA is not available on this system.”)print_gpu_properties()
I could not verify that Jetson Orin NX was even able to use CUDA12 so that might be the most banal answer. The product documentation specifically mentions CUDA10 cores but I have a Ubuntu 22.4 Hostmachine and I dread setting up a new one for Jetpack 5.0