Compute-sanitizer "Error: Target application terminated before first instrumented API call" Ubuntu 22.04

Developer Tools CUDA Developer Tools Compute Sanitizer
1

When running compute-sanitizer ./main, I get the error:

========= COMPUTE-SANITIZER
========= Error: Target application terminated before first instrumented API call
========= Error: couldn't find exit code.

Note that I had to move compute-sanitizer from /usr/bin/compute-sanitizer to /usr/lib/nvidia-cuda-toolkit/compute-sanitizer/compute-sanitizer because otherwise it would crash with the error message Unable to find injection library libsanitizer-collection.so.

I also found the similar issue Compute Sanitizer always report "Target application terminated before first instrumented API call" but I might have a different problem because TreeLauncherSubreaper and TreeLauncherTargetLdPreloadHelper look sufficiently executable:

$ ls -l /usr/bin | grep TreeLauncher
-rwxr-xr-x 1 root root       162648 Nov 19  2021 TreeLauncherSubreaper
-rwxr-xr-x 1 root root        14336 Nov 19  2021 TreeLauncherTargetLdPreloadHelper

Here is the output of strace compute-sanitizer ./main in case it is useful for debugging this issue.

execve("/usr/bin/compute-sanitizer", ["compute-sanitizer", "./main"], 0x7ffe6addd5b8 /* 61 vars */) = 0
brk(NULL)                               = 0x16e6000
arch_prctl(0x3001 /* ARCH_??? */, 0x7ffc390a8b00) = -1 EINVAL (Invalid argument)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x78297bf38000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=108179, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 108179, PROT_READ, MAP_PRIVATE, 3, 0) = 0x78297bf1d000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=21448, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 16424, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x78297bf18000
mmap(0x78297bf19000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x78297bf19000
mmap(0x78297bf1a000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x78297bf1a000
mmap(0x78297bf1b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x78297bf1b000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=14664, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 16440, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x78297bf13000
mmap(0x78297bf14000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x78297bf14000
mmap(0x78297bf15000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x78297bf15000
mmap(0x78297bf16000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x78297bf16000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=14432, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 16424, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x78297bf0e000
mmap(0x78297bf0f000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x78297bf0f000
mmap(0x78297bf10000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x78297bf10000
mmap(0x78297bf11000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x78297bf11000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libutil.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=14432, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 16424, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x78297bf09000
mmap(0x78297bf0a000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x78297bf0a000
mmap(0x78297bf0b000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x78297bf0b000
mmap(0x78297bf0c000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x78297bf0c000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=940560, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 942344, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x78297be22000
mmap(0x78297be30000, 507904, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xe000) = 0x78297be30000
mmap(0x78297beac000, 372736, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8a000) = 0x78297beac000
mmap(0x78297bf07000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xe4000) = 0x78297bf07000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=125488, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 127720, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x78297be02000
mmap(0x78297be05000, 94208, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x78297be05000
mmap(0x78297be1c000, 16384, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1a000) = 0x78297be1c000
mmap(0x78297be20000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1d000) = 0x78297be20000
close(3)                                = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x78297be00000
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\237\2\0\0\0\0\0"..., 832) = 832
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
pread64(3, "\4\0\0\0 \0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0"..., 48, 848) = 48
pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0I\17\357\204\3$\f\221\2039x\324\224\323\236S"..., 68, 896) = 68
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=2220400, ...}, AT_EMPTY_PATH) = 0
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
mmap(NULL, 2264656, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x78297ba00000
mprotect(0x78297ba28000, 2023424, PROT_NONE) = 0
mmap(0x78297ba28000, 1658880, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x28000) = 0x78297ba28000
mmap(0x78297bbbd000, 360448, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1bd000) = 0x78297bbbd000
mmap(0x78297bc16000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x215000) = 0x78297bc16000
mmap(0x78297bc1c000, 52816, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x78297bc1c000
close(3)                                = 0
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x78297bdfd000
arch_prctl(ARCH_SET_FS, 0x78297bdfd780) = 0
set_tid_address(0x78297bdfda50)         = 67846
set_robust_list(0x78297bdfda60, 24)     = 0
rseq(0x78297bdfe120, 0x20, 0, 0x53053053) = 0
mprotect(0x78297bc16000, 16384, PROT_READ) = 0
mprotect(0x78297be20000, 4096, PROT_READ) = 0
mprotect(0x78297bf07000, 4096, PROT_READ) = 0
mprotect(0x78297bf0c000, 4096, PROT_READ) = 0
mprotect(0x78297bf11000, 4096, PROT_READ) = 0
mprotect(0x78297bf16000, 4096, PROT_READ) = 0
mprotect(0x78297bf1b000, 4096, PROT_READ) = 0
mprotect(0xaae000, 225280, PROT_READ)   = 0
mprotect(0x78297bf72000, 8192, PROT_READ) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
munmap(0x78297bf1d000, 108179)          = 0
getrandom("\x9c\xe3\xbd\x54\x76\xaa\xc4\x23", 8, GRND_NONBLOCK) = 8
brk(NULL)                               = 0x16e6000
brk(0x1707000)                          = 0x1707000
futex(0xaf8d9c, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0xaf8da8, FUTEX_WAKE_PRIVATE, 2147483647) = 0
brk(0x1738000)                          = 0x1738000
newfstatat(1, "", {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0x4), ...}, AT_EMPTY_PATH) = 0
write(1, "========= COMPUTE-SANITIZER\n", 28) = 28
stat("./main", {st_mode=S_IFREG|0775, st_size=807664, ...}) = 0
getcwd("/home/username/cuda", 260)    = 22
readlink("/proc/self/exe", "/usr/lib/nvidia-cuda-toolkit/com"..., 260) = 64
stat("/usr/lib/nvidia-cuda-toolkit/compute-sanitizer/libsanitizer-collection.so", {st_mode=S_IFREG|0644, st_size=7778136, ...}) = 0
getcwd("/home/username/cuda", 260)    = 22
access("/home/username/cuda/./main", F_OK) = 0
access("/home/username/cuda/./main", R_OK|X_OK) = 0
epoll_create1(EPOLL_CLOEXEC)            = 3
mkdir("/tmp/NVIDIA-treetracker-xLZxuw", 0700) = 0
access("/home/username/cuda", F_OK)   = 0
stat("/home/username/cuda", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
gettid()                                = 67846
getpid()                                = 67846
openat(AT_FDCWD, "/proc/67846/status", O_RDONLY) = 4
newfstatat(4, "", {st_mode=S_IFREG|0444, st_size=0, ...}, AT_EMPTY_PATH) = 0
read(4, "Name:\tcompute-sanitiz\nUmask:\t000"..., 1024) = 1024
close(4)                                = 0
readlink("/proc/self/exe", "/usr/lib/nvidia-cuda-toolkit/com"..., 4096) = 64
openat(AT_FDCWD, "/usr/lib/nvidia-cuda-toolkit/compute-sanitizer//nvlog.config", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/username/nvlog.config", O_RDONLY) = -1 ENOENT (No such file or directory)
mknodat(AT_FDCWD, "/tmp/NVIDIA-treetracker-xLZxuw/pipe-0-ownerReadingStream", S_IFIFO|0666) = 0
mknodat(AT_FDCWD, "/tmp/NVIDIA-treetracker-xLZxuw/pipe-0-ownerWritingStream", S_IFIFO|0666) = 0
openat(AT_FDCWD, "/tmp/NVIDIA-treetracker-xLZxuw/pipe-0-ownerReadingStream", O_RDONLY|O_NONBLOCK) = 4
openat(AT_FDCWD, "/tmp/NVIDIA-treetracker-xLZxuw/pipe-0-ownerWritingStream", O_RDWR|O_NONBLOCK) = 5
epoll_ctl(3, EPOLL_CTL_ADD, 4, {events=EPOLLIN|EPOLLET, data={u32=24117184, u64=24117184}}) = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x78297bdfda50) = 67847
rt_sigaction(SIGRT_1, {sa_handler=0x78297ba91870, sa_mask=[], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x78297ba42520}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
mmap(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x78297b000000
mprotect(0x78297b001000, 8388608, PROT_READ|PROT_WRITE) = 0
rt_sigprocmask(SIG_BLOCK, ~[], [], 8)   = 0
clone3({flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, child_tid=0x78297b800910, parent_tid=0x78297b800910, exit_signal=0, stack=0x78297b000000, stack_size=0x7ffec0, tls=0x78297b800640} => {parent_tid=[67848]}, 88) = 67848
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
mmap(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x78297a600000
mprotect(0x78297a601000, 8388608, PROT_READ|PROT_WRITE) = 0
rt_sigprocmask(SIG_BLOCK, ~[], [], 8)   = 0
clone3({flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, child_tid=0x78297ae00910, parent_tid=0x78297ae00910, exit_signal=0, stack=0x78297a600000, stack_size=0x7ffec0, tls=0x78297ae00640} => {parent_tid=[67849]}, 88) = 67849
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
mmap(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x782979c00000
mprotect(0x782979c01000, 8388608, PROT_READ|PROT_WRITE) = 0
rt_sigprocmask(SIG_BLOCK, ~[], [], 8)   = 0
clone3({flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, child_tid=0x78297a400910, parent_tid=0x78297a400910, exit_signal=0, stack=0x782979c00000, stack_size=0x7ffec0, tls=0x78297a400640} => {parent_tid=[67850]}, 88) = 67850
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
mmap(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x782979200000
mprotect(0x782979201000, 8388608, PROT_READ|PROT_WRITE) = 0
rt_sigprocmask(SIG_BLOCK, ~[], [], 8)   = 0
clone3({flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, child_tid=0x782979a00910, parent_tid=0x782979a00910, exit_signal=0, stack=0x782979200000, stack_size=0x7ffec0, tls=0x782979a00640} => {parent_tid=[67851]}, 88) = 67851
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=67847, si_uid=1000, si_status=255, si_utime=0, si_stime=0} ---
mmap(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x782978800000
mprotect(0x782978801000, 8388608, PROT_READ|PROT_WRITE) = 0
rt_sigprocmask(SIG_BLOCK, ~[], [], 8)   = 0
clone3({flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, child_tid=0x782979000910, parent_tid=0x782979000910, exit_signal=0, stack=0x782978800000, stack_size=0x7ffec0, tls=0x782979000640} => {parent_tid=[67852]}, 88) = 67852
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
eventfd2(0, EFD_CLOEXEC|EFD_NONBLOCK)   = 6
epoll_create1(EPOLL_CLOEXEC)            = 7
timerfd_create(CLOCK_MONOTONIC, TFD_CLOEXEC) = 8
epoll_ctl(7, EPOLL_CTL_ADD, 6, {events=EPOLLIN|EPOLLERR|EPOLLET, data={u32=24116376, u64=24116376}}) = 0
write(6, "\1\0\0\0\0\0\0\0", 8)         = 8
epoll_ctl(7, EPOLL_CTL_ADD, 8, {events=EPOLLIN|EPOLLERR, data={u32=24116388, u64=24116388}}) = 0
futex(0x1702a48, FUTEX_WAKE_PRIVATE, 1) = 1
clock_gettime(CLOCK_REALTIME, {tv_sec=1727984965, tv_nsec=622728336}) = 0
write(1, "========= Error: Target applicat"..., 82) = 82
futex(0x7ffc390a87c4, FUTEX_WAIT_PRIVATE, 8, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffc390a87c4, FUTEX_WAIT_PRIVATE, 12, NULL) = 0
epoll_ctl(7, EPOLL_CTL_MOD, 6, {events=EPOLLIN|EPOLLERR|EPOLLET, data={u32=24116376, u64=24116376}}) = 0
futex(0x17029e8, FUTEX_WAKE_PRIVATE, 1) = 1
close(7)                                = 0
close(8)                                = 0
close(6)                                = 0
futex(0x1703e88, FUTEX_WAKE_PRIVATE, 2147483647) = 1
futex(0x16ff4b8, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x16ff468, FUTEX_WAKE_PRIVATE, 1) = 0
munmap(0x78297a600000, 8392704)         = 0
close(4)                                = 0
close(5)                                = 0
unlink("/tmp/NVIDIA-treetracker-xLZxuw/pipe-0-ownerReadingStream") = 0
unlink("/tmp/NVIDIA-treetracker-xLZxuw/pipe-0-ownerWritingStream") = 0
close(3)                                = 0
lstat("/tmp/NVIDIA-treetracker-xLZxuw", {st_mode=S_IFDIR|0700, st_size=40, ...}) = 0
openat(AT_FDCWD, "/tmp/NVIDIA-treetracker-xLZxuw", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
newfstatat(3, "", {st_mode=S_IFDIR|0700, st_size=40, ...}, AT_EMPTY_PATH) = 0
getdents64(3, 0x17180d0 /* 2 entries */, 32768) = 48
getdents64(3, 0x17180d0 /* 0 entries */, 32768) = 0
close(3)                                = 0
rmdir("/tmp/NVIDIA-treetracker-xLZxuw") = 0
exit_group(-1)                          = ?
+++ exited with 255 +++

Here is the CUDA code I used, but it happens with every program.

#include <cuda.h>
#include <stdio.h>

__global__ void illegal_read(int *ptr){
    int value = ptr[threadIdx.x];
    printf("%d: %d\n", threadIdx.x, value);
}

int main(){
    int *array;
    int n = 16;

    cudaMallocManaged(&array, n * sizeof(int));

    for(int i = 0; i < n; i++){
        array[i] = i;
    }

    illegal_read<<<1, 32>>>(array);

    cudaDeviceSynchronize();
    cudaFree(array);
    return 0;
}

$ compute-sanitizer --version
NVIDIA (R) Compute Sanitizer
Copyright (c) 2020-2021 NVIDIA Corporation
Version 2021.3.1

$ nvidia-smi
Thu Oct  3 21:26:47 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.107.02             Driver Version: 550.107.02     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3060 ...    Off |   00000000:01:00.0  On |                  N/A |
| N/A   42C    P8             11W /   80W |     252MiB /   6144MiB |     47%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      1331      G   /usr/lib/xorg/Xorg                            149MiB |
+-----------------------------------------------------------------------------------------+

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Thu_Nov_18_09:45:30_PST_2021
Cuda compilation tools, release 11.5, V11.5.119
Build cuda_11.5.r11.5/compiler.30672275_0
2

Can you try with compute-sanitizer from the latest CUDA toolkit? This version seems to 3 years old. You can download it at: CUDA Toolkit 12.6 Update 2 Downloads | NVIDIA Developer

3

I tried the following configurations:

Linux, x86_64, Ubuntu, 22.04, deb (local)

Did not seem to install anything, at least all versions are still the same, although sudo apt-get -y install cuda-toolkit-12-6 now says cuda-toolkit-12-6 is already the newest version (12.6.2-1)

Linux, x86_64, Ubuntu, 22.04, deb (network)

Same as above.

Linux, x86_64, Ubuntu, 22.04, runfile (local)

Failed with

Extraction failed.
Ensure there is enough space in /tmp and that the installation package is not corrupt
Signal caught, cleaning up

/tmp still has over 6.6 GB of free space.

I think I followed these instructions CUDA Installation Guide for Linux previously.

4

Hi, @X100

Can you try to uninstall the toolkit on your machines firstly and then install the latest one ?

5

Well, this took a while. When attempting to remove the old drivers, my system was FUBARed, so I reinstalled a fresh Lubuntu 24.04 and encountered a bunch of bugs:

Runfile missing dependency

The runfile hosted at CUDA Toolkit 12.6 Update 2 Downloads | NVIDIA Developer crashes when gcc is not installed. I’d suggest to add a line like sudo apt install build-essential or similar to the installation instructions.

Runfile crashes with nonsensical error

The runfile crashes when trying to install it. The error message is here:

[INFO]: Driver not installed.
[INFO]: Checking compiler version...
[INFO]: gcc location: /usr/bin/gcc

[INFO]: gcc version: gcc version 13.2.0 (Ubuntu 13.2.0-23ubuntu4) 

[INFO]: Initializing menu
[INFO]: nvidia-fs.setKOVersion(2.22.3)
[INFO]: Setup complete
[INFO]: Installing: Driver
[INFO]: Installing: 560.35.03
[INFO]: Executing NVIDIA-Linux-x86_64-560.35.03.run --ui=none --no-questions --accept-license --disable-nouveau --no-cc-version-check --install-libglvnd  2>&1
[INFO]: Finished with code: 3840
[ERROR]: Install of driver component failed. Consult the driver log at /var/log/nvidia-installer.log for more details.
[ERROR]: Install of 560.35.03 failed, quitting

It is suggested to check the file /var/log/nvidia-installer.log for more details, but that file does not exist:

$ cat /var/log/nvidia-installer.log
cat: /var/log/nvidia-installer.log: No such file or directory

So in the end, I abandoned the runfile installer and went with the network installer instead: CUDA Toolkit 12.6 Update 2 Downloads | NVIDIA Developer

Bad UI in MOK Info

When following the instructions at CUDA Toolkit 12.6 Update 2 Downloads | NVIDIA Developer and installing the open source driver, there will eventually be a screen in blue informing the user that they have to enroll a MOK later. To continue, the screen has to be confirmed with <OK>, but space, enter, mouse and keyboard mashing did not select the <OK> button until I eventually hit the right-arrow key, which then worked. I suggest to add other keys to advance to that button or at least note somewhere that you have to press the right-arrow key.

Lubuntu did not boot after enrolling MOK

After enrolling the MOK, the system would not boot (dual booting Windows 10, if that matters). In GRUB, I could edit the boot settings by pressing e when Lubuntu was selected to remove quiet splash from the second-last or so line, which made the system boot again.

That change could be made permanent by changing the line GRUB_CMDLINE_LINUX_DEFAULT="quiet splash" to GRUB_CMDLINE_LINUX_DEFAULT="" in /etc/default/grub and then running update-grub.

Failed to start nvidia-persistence?ervice

The boot log shows the message nvidia-persistence?ervice [sic]. The first half of the name is red. Not sure what that is or if that is an issue or not. I did not notice any problems otherwise.

/usr/local/cuda-12.6/bin/nvcc and /usr/local/cuda-12.6/bin/compute-sanitizer missing from PATH

nvcc and compute-sanitizer have not been added to PATH by the installer. Not sure if that is intended, but if it is, it would be nice if the installer printed where it is going to install things so I do not have to search for them. They could be located with

sudo apt install plocate
locate nvcc
locate computer-sanitizer

But in the end, compute-sanitizer works properly now, so I did not waste four hours of my precious weekend for nothing.

6

Hi, @X100

Thanks for updating so many details and sorry for the trouble you met. I believe most issue you mentioned can find solution in CUDA Installation Guide for Linux

Glad to know that sanitizer can work for you now. Also if you think CUDA installation need to improve, feel free to raise a topic in CUDA Setup and Installation - NVIDIA Developer Forums

7

I wish I had the time to do that.

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.