CUDA NOT WORKING

Accelerated Computing CUDA CUDA Setup and Installation
1

Im using a ppc64le pc with almalinux 8.5
This is my output of nvidia-smi :
±--------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.23.06 Driver Version: 545.23.06 CUDA Version: 12.3 |
|-----------------------------------------±---------------------±---------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla V100-SXM2-16GB Off | 00000004:04:00.0 Off | 0 |
| N/A 37C P0 51W / 300W | 0MiB / 16384MiB | 0% Default |
| | | N/A |
±----------------------------------------±---------------------±---------------------+
| 1 Tesla V100-SXM2-16GB Off | 00000004:05:00.0 Off | 0 |
| N/A 39C P0 53W / 300W | 0MiB / 16384MiB | 0% Default |
| | | N/A |
±----------------------------------------±---------------------±---------------------+
| 2 Tesla V100-SXM2-16GB Off | 00000035:03:00.0 Off | 0 |
| N/A 35C P0 54W / 300W | 0MiB / 16384MiB | 0% Default |
| | | N/A |
±----------------------------------------±---------------------±---------------------+
| 3 Tesla V100-SXM2-16GB Off | 00000035:04:00.0 Off | 0 |
| N/A 38C P0 51W / 300W | 0MiB / 16384MiB | 3% Default |
| | | N/A |
±----------------------------------------±---------------------±---------------------+

±--------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
±--------------------------------------------------------------------------------------+

this is my nvcc version :

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Fri_Sep__8_19:17:19_PDT_2023
Cuda compilation tools, release 12.3, V12.3.52
Build cuda_12.3.r12.3/compiler.33281558_0

I install driver and cuda with runfile, and I didn’t see anything wrong in /var/log/cuda-installer and nvidia-installer.

When i use cudaGetDeviceCount(&number) i get Initialation error while doing device query test i get :
cudaGetDeviceCount failed – CUDA driver and runtime may be mismatched
FORTRAN STOP

What I am doing wrong?

2

To me, this seems to be a pretty old driver. My system works on version 571.96.
I noticed in my first tests with CUDA that old drivers may prevent your software from actually utilizing the CUDA hardware. I suppose if you installed the driver included in the (also rather old) CUDA Toolkit 12.3 installer, this might be the problem.

Maybe the Driver Installation Guide will help.

3

Have you solved this problem? I have also noticed this issue. If the “persistence mode” is set to “off” after installing the driver, the GPU will not work properly.
Running the CUDA test program appears to run normally, but the output is either incorrect or there is no output at all.