CLBlast fails on context lengths above 2048 after merging #4256

[LostRuins]

Inference with CLBlast fails with a segfault after the commit that merged #4256 on context sizes above 2k when all GPU layers are offloaded.

Command line:
C:\test\llama-b1601-bin-win-clblast-x64>main.exe -m E:\LLaMA\models\airoboros-mistral2.2-7b.Q4_K_S.gguf -c 4096 -b 512 -n 32 -ngl 33 -f C:\test\test.txt

main: build = 1601 (5a7d312)
main: built with MSVC 19.37.32826.1 for x64
main: seed  = 1701534899
ggml_opencl: selecting platform: 'NVIDIA CUDA'
ggml_opencl: selecting device: 'NVIDIA GeForce RTX 2060'
ggml_opencl: device FP16 support: false

Result:
Prompt processing starts, and then segfaults halfway around the 2k token mark, before generation begins. Only if the prompt is short enough (less than 2k tokens) it appears to work.

[ggerganov]

Does it work with this patch:

diff --git a/llama.cpp b/llama.cpp
index fd905ade..69c45c3f 100644
--- a/llama.cpp
+++ b/llama.cpp
@@ -3813,7 +3813,7 @@ static struct ggml_tensor * llm_build_kqv(
     struct ggml_tensor * kq = ggml_mul_mat(ctx, k, q);
     cb(kq, "kq", il);
 
-    if (max_alibi_bias > 0.0f) {
+    if (true) {
         // temporary branch until we figure out how to handle ggml_alibi through ggml_add
         kq = ggml_scale(ctx, kq, kq_scale);
         cb(kq, "kq_scaled", il);

[LostRuins]

Nope, unfortunately this did not fix the issue, it still segfaults around the same point.

[ggerganov]

Hm, I don't see what could have affected the OpenCL backend in that change.
Any extra information that you can provide (e.g. stack trace)? Does it depend on the value of -ngl?

[LostRuins]

There's no stack trace. In fact, there's no printout whatsoever, the program simply halts. I tried it again with 0 layers offloaded and it seems to happen too, it still crashes at the same place. CUDA is fine, however.

Here's a video of b1579 vs b1601, showing the differences. The video has been sped up by 2x, but you can rewind/pause it at any point to review.

demo.mp4

The test text file I used for input is the first 5 sections of the GPL license, which you can find here:
test.txt

I am able to repro this consistently as it crashes at the same place. Reducing the prompt to a shorter one allows it to work.
I am on Windows 10, running with RTX2060.

[AlpinDale]

Can confirm this happens for me too. Same command and prompt as @LostRuins. Hardware is RTX 2070S and Intel i7-8700, and I'm using Linux 6.5.9. Happens with -ngl 0 and -ngl 99. The error I get is:

free(): invalid next size (normal)
zsh: IOT instruction (core dumped)

Different error followed by a segfault with -ngl 32 (7B GGUF model):

ggml_opencl: clSetKernelArg(*to_fp32_cl, 0, sizeof(cl_mem), &d_Q) error -38 at ggml-opencl.cpp:1733

[ggerganov]

And b1600 works?

[AlpinDale]

I tested more, and I get a coredump with lower -c values too. Tried 2048 and 1600.
It's an IOT instruction core dump.

[LostRuins]

Reverting this specific commit: ggml : add ggml_soft_max_ext (#4256) seems to work.

[slaren]

The free error suggests that this is a memory corruption issue. The changes in #4256 are not likely to be related. Running this with an ASAN build (enable LLAMA_SANITIZE_ADDRESS and LLAMA_SANITIZE_UNDEFINED) may show the source of the issue.

[ggerganov]

I'm able to reproduce - looking into it

[AlpinDale]

I built with ASan, here's the error traceback I get when running with the command:

./main -m ~/models/openhermes-2-mistral-7b.Q6_K.gguf -c 4096 -b 512 -n 32 -ngl 99 -f test.txt

Error:

Log start
main: build = 1604 (33e171d)
main: built with cc (GCC) 13.2.1 20230801 for x86_64-pc-linux-gnu
main: seed  = 1701609933
ggml_opencl: clGetPlatformIDs(NPLAT, platform_ids, &n_platforms) error -1001 at ggml-opencl.cpp:965

=================================================================
==2203735==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 264 byte(s) in 1 object(s) allocated from:
    #0 0x7f12666e0cc1 in __interceptor_calloc /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:77
    #1 0x7f1260a5dee0  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x205dee0)
    #2 0x7f12608e566a  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x1ee566a)
    #3 0x7f1265c02fd4  (/opt/cuda/lib64/libOpenCL.so.1+0x2fd4) (BuildId: b3217362255db6f1188e7596454ffe8bc4606b53)

Direct leak of 56 byte(s) in 1 object(s) allocated from:
    #0 0x7f12666e0cc1 in __interceptor_calloc /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:77
    #1 0x7f1260a5b80a  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x205b80a)
    #2 0x7f1260a5cfaa  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x205cfaa)
    #3 0x7f12608e566a  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x1ee566a)
    #4 0x7f1265c02fd4  (/opt/cuda/lib64/libOpenCL.so.1+0x2fd4) (BuildId: b3217362255db6f1188e7596454ffe8bc4606b53)

Direct leak of 56 byte(s) in 1 object(s) allocated from:
    #0 0x7f12666e0cc1 in __interceptor_calloc /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:77
    #1 0x7f1260a5b8ae  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x205b8ae)
    #2 0x7f1260a5cfaa  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x205cfaa)
    #3 0x7f12608e566a  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x1ee566a)
    #4 0x7f1265c02fd4  (/opt/cuda/lib64/libOpenCL.so.1+0x2fd4) (BuildId: b3217362255db6f1188e7596454ffe8bc4606b53)

Direct leak of 56 byte(s) in 1 object(s) allocated from:
    #0 0x7f12666e0cc1 in __interceptor_calloc /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:77
    #1 0x7f1260a5b936  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x205b936)
    #2 0x7f1260a5cfaa  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x205cfaa)
    #3 0x7f12608e566a  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x1ee566a)
    #4 0x7f1265c02fd4  (/opt/cuda/lib64/libOpenCL.so.1+0x2fd4) (BuildId: b3217362255db6f1188e7596454ffe8bc4606b53)

Direct leak of 32 byte(s) in 1 object(s) allocated from:
    #0 0x7f12666e1359 in __interceptor_malloc /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x7f1260b70de2  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x2170de2)
    #2 0x7f1260a576bf  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x20576bf)
    #3 0x7f1260a5d18f  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x205d18f)
    #4 0x7f12608e566a  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x1ee566a)
    #5 0x7f1265c02fd4  (/opt/cuda/lib64/libOpenCL.so.1+0x2fd4) (BuildId: b3217362255db6f1188e7596454ffe8bc4606b53)

Direct leak of 32 byte(s) in 1 object(s) allocated from:
    #0 0x7f12666e1359 in __interceptor_malloc /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x7f1260b70de2  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x2170de2)
    #2 0x7f1260a576db  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x20576db)
    #3 0x7f1260a5d18f  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x205d18f)
    #4 0x7f12608e566a  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x1ee566a)
    #5 0x7f1265c02fd4  (/opt/cuda/lib64/libOpenCL.so.1+0x2fd4) (BuildId: b3217362255db6f1188e7596454ffe8bc4606b53)

Indirect leak of 320 byte(s) in 1 object(s) allocated from:
    #0 0x7f12666e0cc1 in __interceptor_calloc /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:77
    #0 0x7f12666e0cc1 in __interceptor_calloc /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:77
    #1 0x7f1260b70df9  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x2170df9)
    #2 0x7f1260a576db  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x20576db)
    #3 0x7f1260a5d18f  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x205d18f)
    #4 0x7f12608e566a  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x1ee566a)
    #5 0x7f1265c02fd4  (/opt/cuda/lib64/libOpenCL.so.1+0x2fd4) (BuildId: b3217362255db6f1188e7596454ffe8bc4606b53)

Indirect leak of 320 byte(s) in 1 object(s) allocated from:
    #0 0x7f12666e0cc1 in __interceptor_calloc /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:77
    #1 0x7f1260b70df9  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x2170df9)
    #2 0x7f1260a576bf  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x20576bf)
    #3 0x7f1260a5d18f  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x205d18f)
    #4 0x7f12608e566a  (/home/alpindale/AI-Stuff/tools/llama.cpp/main+0x1ee566a)
    #5 0x7f1265c02fd4  (/opt/cuda/lib64/libOpenCL.so.1+0x2fd4) (BuildId: b3217362255db6f1188e7596454ffe8bc4606b53)

SUMMARY: AddressSanitizer: 1136 byte(s) leaked in 8 allocation(s).

Reverted ef47ec18da469423c276b683dd9b5741cee7023e (#4256) and re-trying now.