`Libdl.dlopen` doesn't find shared libraries anymore

[nsmith5]

Libdl.dlopen doesn't seem to be able to find shared libraries the way it did on 0.6.x

On Julia 0.6.x

julia> versioninfo()
Julia Version 0.6.2
Commit d386e40c17 (2017-12-13 18:08 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Prescott)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, broadwell)

julia> lib = Libdl.dlopen("libz")
Ptr{Void} @0x00007f47adcb91a0

On Julia 0.7.x-DEV

julia> versioninfo()
Julia Version 0.7.0-DEV.4631
Commit 9a55c8fbc* (2018-03-19 03:59 UTC)
Platform Info:
  OS: Linux (x86_64-redhat-linux)
  CPU: Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, skylake)
Environment:

julia> using Libdl

julia> lib = Libdl.dlopen("libz")
ERROR: could not load library "libz"
libz.so: cannot open shared object file: No such file or directory
Stacktrace:
 [1] dlopen(::String) at /builddir/build/BUILD/julia/build/usr/share/julia/site/v0.7/Libdl/src/Libdl.jl:99
 [2] top-level scope
 [3] macro expansion at /builddir/build/BUILD/julia/build/usr/share/julia/site/v0.7/REPL/src/REPL.jl:117 [inlined]
 [4] (::getfield(REPL, Symbol("##28#29")){REPL.REPLBackend})() at ./event.jl:92

julia> lib = Libdl.dlopen("libz.so")
ERROR: could not load library "libz.so"
libz.so: cannot open shared object file: No such file or directory
Stacktrace:
 [1] dlopen(::String) at /builddir/build/BUILD/julia/build/usr/share/julia/site/v0.7/Libdl/src/Libdl.jl:99
 [2] top-level scope
 [3] macro expansion at /builddir/build/BUILD/julia/build/usr/share/julia/site/v0.7/REPL/src/REPL.jl:117 [inlined]
 [4] (::getfield(REPL, Symbol("##28#29")){REPL.REPLBackend})() at ./event.jl:92

julia> lib = Libdl.dlopen("libz.so.1")
Ptr{Nothing} @0x00007f139153b520

As you can see, it works once I specify the exact name of the shared library, but not before then.

Other Details

• Operating system: Linux (Fedora 27)

[vtjnash]

Yes, you should either (1) use BinDeps (2) specify the name of the correct library version (3) install the libz-dev package. This was changed intentionally to reduce the number of test configurations, since many systems (Windows, Nix, etc.) require you to specify the correct name in full.

[StefanKarpinski]

Seems likely to be due to #22828.

[StefanKarpinski]

Reopening because I think this is the canary in the coal mine and we're going to get a lot of people hitting this problem with no clue what to do and there should be some better way of helping them.

[nsmith5]

Yeah, I didn't spot anything in NEWS.md about the behaviour change and didn't find anything relevant in a quick search through the PR's and issues. I suspect a lot of package breakage will occur because of this (This example comes from CodecZlib in fact..) and documented alternatives will be very helpful.

[nsmith5]

Out of curiousity, why does installing the header files (eg. libz-dev / libz-devel in this case) fix this issue?

[nsmith5]

It seems like many of the examples of ccall from the documentation will probably need to be updated. This first example from Calling C and Fortran Code for instance:

julia> t = ccall((:clock, "libc"), Int32, ())
ERROR: error compiling top-level scope: could not load library "libc"
/usr/bin/../lib64/libc.so: invalid ELF header
Stacktrace:
 [1] macro expansion at /builddir/build/BUILD/julia/build/usr/share/julia/site/v0.7/REPL/src/REPL.jl:117 [inlined]
 [2] (::getfield(REPL, Symbol("##28#29")){REPL.REPLBackend})() at ./event.jl:92

julia> t = ccall((:clock, "/usr/lib64/libc.so.6"), Int32, ())
2667541

[StefanKarpinski]

This is a breaking change and should have a deprecation to let people know what they should do.

[vtjnash]

I suspect a lot of package breakage will occur because of this

Not very many. CodecZlib is one of the only libraries that doesn't use BinDeps. And it'll force them to fix their package for use on less ubiquitous platforms like NixOS (instead of making the fix conditional on running on Windows). And force us to fix our documentation (while sometimes true that these functions are in a shlib named libc, that's not particularly universal).

[StefanKarpinski]

This not only needs docs and NEWS, I really think we should not be just breaking this but providing a proper deprecation. We'll see how it goes but I think this just the first of many complaints we'll see.

[staticfloat]

Reading through the linked PR, I disagree that this should have been merged. Yes, there are reasons for wanting to encourage users to use versioned library names when opening them, but we don't force users to specify a major and minor version number of a Julia package when they load it, we just load the latest available and provide the option (through Pkg.pin() or whatever) to specify if they have the need to restrict what they load. I think the same argument could be made regarding dynamic libraries.