Using CUDA in l4t-cuda Docker container

[bill.mania]

I have a Jetson TX2 with:

# uname -a
Linux 4f129e5e8c83 4.9.253-tegra #3 SMP PREEMPT Thu Nov 18 16:06:09 EST 2021 aarch64 aarch64 aarch64 GNU/Linux

# cat /etc/issue
Ubuntu 18.04.5 LTS \n \l

# ls -l /usr/local/cuda
lrwxrwxrwx 1 root root 20 Jul 23  2021 /usr/local/cuda -> /usr/local/cuda-10.2

# dpkg --list | grep tegra

ii  cti-l4t-utils                                 4.9.253-tegra-32.6.1-20211119-CTI-TX2-V001 arm64        Connect Tech Jetson Utility Package
ii  nvidia-l4t-kernel                             4.9.253-tegra-32.6.1-20211119-CTI-TX2-V001 arm64        Connect Tech Jetson TX2 Kernel Package
ii  nvidia-l4t-kernel-dtbs                        4.9.253-tegra-32.6.1-20211119-CTI-TX2-V001 arm64        Connect Tech Jetson DTB Package
ii  nvidia-l4t-kernel-headers                     4.9.253-tegra-32.6.1-20211119-CTI-TX2-V001 arm64        Connect Tech Jetson Headers Package

I can successfully compile and execute both deviceQuery and deviceQueryDrv on the base operating system, outside of any Docker container and the output is

$ ./deviceQuery
./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "NVIDIA Tegra X2"
  CUDA Driver Version / Runtime Version          10.2 / 10.2
  CUDA Capability Major/Minor version number:    6.2
  Total amount of global memory:                 7859 MBytes (8240721920 bytes)
  ( 2) Multiprocessors, (128) CUDA Cores/MP:     256 CUDA Cores

If I then build a Docker image using

ARG BUILD_ARCH=aarch64
FROM nvcr.io/nvidia/l4t-cuda:10.2.460-runtime

In that container I can successfully compile the deviceQuery utilities, but executing either of them gives

# ./deviceQuery 
./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 35
-> CUDA driver version is insufficient for CUDA runtime version
Result = FAIL

What am I missing, please?

[AastaLLL]

Hi,

We test the container with r32.7.1 (JetPack4.6.1) and r32.6.1 (JetPack4.6) device.
Both can run the deviceQuery binary correctly.

Not sure if anything different between us.
Below are our testing steps for your reference:

$ cd /usr/local/cuda-10.2/samples/1_Utilities/deviceQuery
$ sudo make
$ mkdir ~/tmp
$ cp /usr/local/cuda-10.2/samples/1_Utilities/deviceQuery/deviceQuery ~/tmp/
$ sudo docker run -it --rm --net=host --runtime nvidia -e DISPLAY=$DISPLAY -v /home/nvidia/tmp:/home/nvidia -v /tmp/.X11-unix/:/tmp/.X11-unix nvcr.io/nvidia/l4t-cuda:10.2.460-runtime

# cd /home/nvidia/
# ./deviceQuery

Thanks.

[bill.mania]

The suggested sudo docker run didn’t work for me - it complained of permission issues.

However, the following command did work:

docker run \
    --privileged \
    -it \
    --rm \
    --net=host \
    --runtime nvidia \
    -v /home/bill/tmp:/home/tmp \
    nvcr.io/nvidia/l4t-cuda:10.2.460-runtime