LLVM mis-optimize due to returntwice function

[maleadt]

One of my packages (CUDAdrv) has recently started failing on julia master, with a segfault in typemap.c. I've bisected this issue to e2bd129 (all backtraces and line numbers below are on that commit's tree). I'm not sure where to start debugging this, so I'm at least reporting it here already.

This causes a segfault in jl_typemap_level_assoc_exact:

signal (11): Segmentation fault
while loading CUDAdrv/test/core.jl, in expression starting on line 172
sig_match_fast at src/gf.c:1707
jl_apply_generic at src/gf.c:1886
Type at CUDAdrv/src/module.jl:67
unknown function (ip: 0x7f6198ad561e)
jl_call_method_internal at src/julia_internal.h:92
jl_apply_generic at src/gf.c:1931
do_call at src/interpreter.c:65
eval at src/interpreter.c:188
eval_body at src/interpreter.c:469
eval_body at src/interpreter.c:515
jl_interpret_call at src/interpreter.c:573
jl_interpret_toplevel_thunk at src/interpreter.c:580
jl_toplevel_eval_flex at src/toplevel.c:543
jl_parse_eval_all at src/ast.c:700
jl_load at src/toplevel.c:566
jl_load_ at src/toplevel.c:575
include_from_node1 at ./loading.jl:426
unknown function (ip: 0x7f639ef3716c)
jl_call_method_internal at src/julia_internal.h:92
jl_apply_generic at src/gf.c:1931
do_call at src/interpreter.c:65
eval at src/interpreter.c:188
jl_interpret_toplevel_expr at src/interpreter.c:31
jl_toplevel_eval_flex at src/toplevel.c:529
jl_parse_eval_all at src/ast.c:700
jl_load at src/toplevel.c:566
jl_load_ at src/toplevel.c:575
include_from_node1 at ./loading.jl:426
unknown function (ip: 0x7f639ef3716c)
jl_call_method_internal at src/julia_internal.h:92
jl_apply_generic at src/gf.c:1931
process_options at ./client.jl:266
_start at ./client.jl:322
unknown function (ip: 0x7f639ef75124)
jl_call_method_internal at src/julia_internal.h:92
jl_apply_generic at src/gf.c:1931
jl_apply at ui/../src/julia.h:1396
true_main at ui/repl.c:546
main at ui/repl.c:674
unknown function (ip: 0x7f63a5013740)
unknown function (ip: 0x401818)
Allocations: 1748036 (Pool: 1746766; Other: 1270); GC: 4
Allocations: 1748036 (Pool: 1746766; Other: 1270); GC: 4

Running in GDB makes it segfault somewhere else, but I assume due to the same problem (jl_typeof(NULL)):

Thread 1 "julia" received signal SIGSEGV, Segmentation fault.
0x00007ffff76c9407 in jl_typemap_level_assoc_exact (cache=0x7ffdf1802950, args=0x7fffffffae90, n=3, offs=1 '\001') at src/typemap.c:788
788         jl_value_t *ty = (jl_value_t*)jl_typeof(a1);


(gdb) l
783 
784 jl_typemap_entry_t *jl_typemap_level_assoc_exact(jl_typemap_level_t *cache, jl_value_t **args, size_t n, int8_t offs)
785 {
786     if (n > offs) {
787         jl_value_t *a1 = args[offs];
788         jl_value_t *ty = (jl_value_t*)jl_typeof(a1);
789         assert(jl_is_datatype(ty));
790         if (ty == (jl_value_t*)jl_datatype_type && cache->targ != (void*)jl_nothing) {
791             union jl_typemap_t ml_or_cache = mtcache_hash_lookup(cache->targ, a1, 1, offs);
792             jl_typemap_entry_t *ml = jl_typemap_assoc_exact(ml_or_cache, args, n, offs+1);

(gdb) call jl_(args[0])
Base.#==()
(gdb) p args[1]
$3 = (jl_value_t *) 0x0
(gdb) call jl_(args[2])
CUDAdrv.CuError(code=209, info=Base.Nullable{String}(isnull=true, value=#<null>))

... with this comparison (against 209 == CUDAdrv.ERROR_NO_BINARY_FOR_GPU) originating from:

try
    @apicall(:cuModuleLoadDataEx,
            (Ptr{CuModule_t}, Ptr{Cchar}, Cuint, Ref{CUjit_option}, Ref{Ptr{Void}}),
            handle_ref, data, length(optionKeys), optionKeys, optionValues)
catch err
    (err == ERROR_NO_BINARY_FOR_GPU || err == ERROR_INVALID_IMAGE) || rethrow(err)
    options = decode(optionKeys, optionValues)
    rethrow(CuError(err.code, options[ERROR_LOG_BUFFER]))
end

I've not been able to reduce the test case, as reduced versions did not reliably trigger the segfault on all my systems anymore, while the full CUDAdrv test suite does. I've tested on two Linux64 systems (one Debian 8, one Arch), with fresh builds without any Makefile flags.

@yuyichao any ideas what might be causing this, or where to look for clues?

[yuyichao]

Error in TLS access should error much more reliably and earlier than this. Does undefing JL_ELF_TLS_VARIANT here get rid of the segfault?

[maleadt]

It does.

[yuyichao]

LLVM seems to be generating wrong asm. I reduced it to something that doesn't need libcuda (load a dummy shared library instead) but I can only get segfault when running it under optirun.

[yuyichao]

OK, with this I could get a segfault outside optirun

immutable CuError
    code::Int
    info::Nullable{String}

    CuError(code) = new(code, Nullable{String}())
end

const libcuda = Ref{Ptr{Void}}(C_NULL)

export CuModuleData

"Create a CUDA module from a string containing PTX code."
function CuModuleData(data)
    handle_ref = Ref{Ptr{Void}}()
    options = Dict{Cint,Any}()
    options[0] = Array(UInt8, 1024*1024)
    optionKeys, optionValues = encode(options)

    try
        throw(CuError(ccall(Libdl.dlsym(libcuda[], :cuModuleLoadDataEx), Cint,
                            (Ptr{Ptr{Void}}, Ptr{Cchar}, Cuint, Ref{Cint}, Ref{Ptr{Void}}), handle_ref, data, length(optionKeys), optionKeys, optionValues)))
    catch err
        err == 1
    end
end

@noinline function encode(options::Dict{Cint,Any})
    return Cint[], Ptr{Void}[]
end

CuModuleData("")

[maleadt]

Thanks; yes this case also segfaults on both aforementioned systems.

[yuyichao]

reproduced with the following C code in clang so this is a llvm bug. The C program should print 0 in the last line but it prints 4 in the last line when compiling with clang 3.8 with any optimization level.

AFAICT what's happening is that the no-return branch (the one ends with longjmp) uses a + 4 many times with enough operations in between so that the value has to be spilled to the stack (there aren't enough callee save register to keep all of them). LLVM incorrectly uses the stack slot for a on this branch (since after all a is not used on this branch). However, this is invalid since setjmp can return twice and the stack store modifies memory that is still live on the other branch which we'll run later.

What's happening in the julia code is basically replacing a with the thread pointer. After #17116, we emit pool address in the generated code directly (thread_pointer + offset) and together with #17178 opens up more opportunity for LLVM to fold the offset calculations (that's the whole point....). However, for the code above, this happens to cause a stack spill (the same pool address is used on the throw branch a few times and therefore llvm decided to spill it as a whole) and it reuses the slot for the original thread pointer just like the C code below, causing the segfault.

//

#include <setjmp.h>
#include <stdio.h>
#include <stdlib.h>

jmp_buf env;

__attribute__((noinline)) int f2(int v)
{
    __asm__ volatile("":::"memory");
    return v * v;
}

__attribute__((noinline)) int f(int a)
{
    int b = random();
    int c = random();
    int d = random();
    int e = random();
    int f = random();
    int g = random();
    int h = random();
    int i = random();
    double k = f2(b) + f2(c + f2(d + f2(e + f2(f + f2(g + f2(h + i))))));
    k *= b;
    k -= c;
    k += i;
    if (setjmp(env) == 0) {
        printf("%d\n", a + 4);
        b = random();
        c = random();
        d = random();
        e = random();
        f = random();
        g = random();
        h = random();
        i = random();
        k = f2(b) + f2(c + f2(d + f2(e + f2(f + f2(g + f2(h + i))))));
        k *= b;
        k -= c;
        k += i;
        printf("%d\n", a + 4);
        b = random();
        c = random();
        d = random();
        e = random();
        f = random();
        g = random();
        h = random();
        i = random();
        k = f2(b) + f2(c + f2(d + f2(e + f2(f + f2(g + f2(h + i))))));
        k *= b;
        k -= c;
        k += i;
        printf("%d\n", a + 4);
        b = random();
        c = random();
        d = random();
        e = random();
        f = random();
        g = random();
        h = random();
        i = random();
        k = f2(b) + f2(c + f2(d + f2(e + f2(f + f2(g + f2(h + i))))));
        k *= b;
        k -= c;
        k += i;
        printf("%d\n", a + 4);
        printf("%f\n", k);
        longjmp(env, 1);
    }
    else {
        printf("%d\n", a);
    }
    return a;
}

int main()
{
    return f(0);
}

[yuyichao]

I'm also pretty sure it is not UB to make a non volatile since it is not modified after setjmp.

[yuyichao]

Report upstream as https://llvm.org/bugs/show_bug.cgi?id=28431. This is likely a problem in isel so I probably can't help much with debugging it further.

[yuyichao]

And the follow julia code reproduces the same issue on 0.4 too.

@noinline f2(v) = v
@noinline p(v) = println(v)
@noinline r() = rand(Int)

function f(a)
    b0 = r()
    c0 = r()
    d0 = r()
    e0 = r()
    f0 = r()
    g0 = r()
    h0 = r()
    i0 = r()
    k::Float64 = f2(b0) + f2(c0 + f2(d0 + f2(e0 + f2(f0 + f2(g0 + f2(h0 + i0))))));
    k *= b0
    k -= c0
    k += i0
    try
        b = r()
        c = r()
        d = r()
        e = r()
        f = r()
        g = r()
        h = r()
        i = r()
        k += f2(b) + f2(c + f2(d + f2(e + f2(f + f2(g + f2(h + i))))));
        k *= b
        k -= c
        k += i
        p(a + 4)
        b = r()
        c = r()
        d = r()
        e = r()
        f = r()
        g = r()
        h = r()
        i = r()
        k += f2(b) + f2(c + f2(d + f2(e + f2(f + f2(g + f2(h + i))))));
        k *= b
        k -= c
        k += i
        p(a + 4)
        b = r()
        c = r()
        d = r()
        e = r()
        f = r()
        g = r()
        h = r()
        i = r()
        k += f2(b) + f2(c + f2(d + f2(e + f2(f + f2(g + f2(h + i))))));
        k *= b
        k -= c
        k += i
        p(a + 4)
        b = r()
        c = r()
        d = r()
        e = r()
        f = r()
        g = r()
        h = r()
        i = r()
        k += f2(b) + f2(c + f2(d + f2(e + f2(f + f2(g + f2(h + i))))));
        k *= b
        k -= c
        k += i
        p(a + 4)
        b = r()
        c = r()
        d = r()
        e = r()
        f = r()
        g = r()
        h = r()
        i = r()
        k += f2(b) + f2(c + f2(d + f2(e + f2(f + f2(g + f2(h + i))))));
        k *= b
        k -= c
        k += i
        p(a + 4)
        b = r()
        c = r()
        d = r()
        e = r()
        f = r()
        g = r()
        h = r()
        i = r()
        k += f2(b) + f2(c + f2(d + f2(e + f2(f + f2(g + f2(h + i))))));
        k *= b
        k -= c
        k += i
        p(a + 4)
        throw(k)
    catch
        p(a)
    end
end

# @code_llvm f(0)
# @code_native f(0)
f(0)

[maleadt]

Impressive sleuthing, thanks for looking into this!
Any workarounds for now? LLVM 27190 doesn't seem terribly active.

[yuyichao]

LLVM #27190 doesn't seem terribly active.

It's more active than many other llvm issues I've seen.

Any workarounds for now?

Not really. Maybe try to move something to other functions to keep the register pressure low in the function with try-catch?

[yuyichao]

Also it's probably better to check the return status directly instead of throwing it eagerly and rethrow it after checking it's type