Can't run Docker container with CUDA >=11.8

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
1

When trying starting docker container with CUDA 11.8 the following error is shown:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: nvidia-container-runot terminate successfully: exit status 1: time="2023-11-01T18:16:58-03:00" level=error msg="failed to create NVIDIA Container Runtime: failed to cCI spec modifier: requirements not met: unsatisfied condition: cuda>=11.8 (cuda=11.4)"
: unknown.

CUDA 11.8 is installed in my system (using https://developer.nvidia.com/blog/simplifying-cuda-upgrades-for-nvidia-jetson-users/ instructions)

LD_LIBRARY_PATH is set according to tutorial.

Full log output:

lxuser@agxdev:~$ echo $LD_LIBRARY_PATH
/usr/local/cuda-11.8/compat
lxuser@agxdev:~$
lxuser@agxdev:~$
lxuser@agxdev:~$ /usr/local/cuda/bin/nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:43:33_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0
lxuser@agxdev:~$
lxuser@agxdev:~$
lxuser@agxdev:~$ cd cuda-samples/Samples/1_Utilities/deviceQuery
lxuser@agxdev:~/cuda-samples/Samples/1_Utilities/deviceQuery$
lxuser@agxdev:~/cuda-samples/Samples/1_Utilities/deviceQuery$ ./deviceQuery
./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "Orin"
  CUDA Driver Version / Runtime Version          11.8 / 11.8
  CUDA Capability Major/Minor version number:    8.7
  Total amount of global memory:                 62797 MBytes (65847091200 bytes)
  (016) Multiprocessors, (128) CUDA Cores/MP:    2048 CUDA Cores
  GPU Max Clock rate:                            1300 MHz (1.30 GHz)
  Memory Clock rate:                             1300 Mhz
  Memory Bus Width:                              128-bit
  L2 Cache Size:                                 4194304 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total shared memory per multiprocessor:        167936 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  1536
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            Yes
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Managed Memory:                Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 0 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 11.8, CUDA Runtime Version = 11.8, NumDevs = 1
Result = PASS
lxuser@agxdev:~/cuda-samples/Samples/1_Utilities/deviceQuery$
lxuser@agxdev:~/cuda-samples/Samples/1_Utilities/deviceQuery$ cd
lxuser@agxdev:~$
lxuser@agxdev:~$ docker run -it --rm --runtime nvidia nvidia/cuda:11.8.0-devel-ubuntu22.04
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: nvidia-container-runtime did not terminate successfully: exit status 1: time="2023-11-01T18:34:14-03:00" level=error msg="failed to create NVIDIA Container Runtime: failed to construct OCI spec modifier: requirements not met: unsatisfied condition: cuda>=11.8 (cuda=11.4)"
: unknown.
lxuser@agxdev:~$ echo $LD_LIBRARY_PATH
/usr/local/cuda-11.8/compat
lxuser@agxdev:~$ ls $LD_LIBRARY_PATH
libcuda.so  libcuda.so.1  libcuda.so.1.1  libnvidia-nvvm.so  libnvidia-nvvm.so.4  libnvidia-ptxjitcompiler.so  libnvidia-ptxjitcompiler.so.1
lxuser@agxdev:~$

I have also tried to add the LD_LIBRARY_PATH settings in docker systemd unit environment but no luck. Any Hints?

Tegra release/JP version:
# R35 (release), REVISION: 2.1 , GCID: 32413640 , BOARD: t186ref, EABI: aarch64, DATE: Tue Jan 24 23 : 38 : 33 UTC 2023
Package: nvidia-jetpack
Version: 5.1 -b147
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-jetpack-runtime (= 5.1 -b147), nvidia-jetpack-dev (= 5.1 -b147)

3

@evandro.hauenstein this container is not built for Jetson/JetPack - instead, please use l4t-jetpack or l4t-cuda containers, or use l4t-base and install your desired version of CUDA into it.

1 Like
5

Thank you very much @dusty_nv

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.