Homebrew's Julia Build is not built correctly and causes package precompilation errors

[SvenDuve]

Hello,

I can't precompile DifferentialEquations pointing me to an issue with LinearSolve, and several other packages( Also not pre-compiling: StochasticDiffEq.jl, OrdinaryDiffEq.jl).

I was trying to add the package standalone, and got the following:

julia> using LinearSolve │ Package LinearSolve not found, but a package named LinearSolve is available │ from a registry. │ Install package? │ (@v1.8) pkg> add LinearSolve └ (y/n/o) [y]: y Updating registry at /.julia/registries/General.tomlResolving package versions... Updating/.julia/environments/v1.8/Project.toml[7ed4a6bd] + LinearSolve v1.35.0 No Changes to~/.julia/environments/v1.8/Manifest.toml`
[ Info: Precompiling LinearSolve [7ed4a6bd-45f5-4d41-b270-4a48e9bafcae]
ERROR: LoadError: MethodError: convert(::Type{Union{}}, ::SparseArrays.SparseMatrixCSC{Float64, Int64}) is ambiguous. Candidates:
convert(T::Type{<:LinearAlgebra.Bidiagonal}, m::AbstractMatrix) in LinearAlgebra at /opt/homebrew/Cellar/julia/1.8.5/share/julia/stdlib/v1.8/LinearAlgebra/src/bidiag.jl:203
convert(T::Type{<:SparseArrays.AbstractSparseMatrixCSC}, m::AbstractMatrix) in SparseArrays at /opt/homebrew/Cellar/julia/1.8.5/share/julia/stdlib/v1.8/SparseArrays/src/sparsematrix.jl:745
convert(T::Type{<:LinearAlgebra.SymTridiagonal}, m::SparseArrays.AbstractSparseMatrixCSC) in SparseArrays at /opt/homebrew/Cellar/julia/1.8.5/share/julia/stdlib/v1.8/SparseArrays/src/sparsematrix.jl:749
convert(T::Type{<:LinearAlgebra.LowerTriangular}, m::SparseArrays.AbstractSparseMatrixCSC) in SparseArrays at /opt/homebrew/Cellar/julia/1.8.5/share/julia/stdlib/v1.8/SparseArrays/src/sparsematrix.jl:753
convert(T::Type{<:SparseArrays.SparseVector}, m::SparseArrays.AbstractSparseMatrixCSC) in SparseArrays at /opt/homebrew/Cellar/julia/1.8.5/share/julia/stdlib/v1.8/SparseArrays/src/sparsevector.jl:466
convert(T::Type{<:LinearAlgebra.Tridiagonal}, m::SparseArrays.AbstractSparseMatrixCSC) in SparseArrays at /opt/homebrew/Cellar/julia/1.8.5/share/julia/stdlib/v1.8/SparseArrays/src/sparsematrix.jl:751
convert(T::Type{<:LinearAlgebra.UpperTriangular}, m::SparseArrays.AbstractSparseMatrixCSC) in SparseArrays at /opt/homebrew/Cellar/julia/1.8.5/share/julia/stdlib/v1.8/SparseArrays/src/sparsematrix.jl:755
convert(T::Type{<:LinearAlgebra.Diagonal}, m::SparseArrays.AbstractSparseMatrixCSC) in SparseArrays at /opt/homebrew/Cellar/julia/1.8.5/share/julia/stdlib/v1.8/SparseArrays/src/sparsematrix.jl:747
convert(::Type{Union{}}, a::AbstractArray) in Base at array.jl:618
convert(::Type{SA}, a::AbstractArray) where SA<:StaticArraysCore.StaticArray in StaticArrays at /Users/svenduve/.julia/packages/StaticArrays/jA1zK/src/convert.jl:194
convert(::Type{T}, a::AbstractArray) where T<:Array in Base at array.jl:617
convert(T::Type{<:BitArray}, a::AbstractArray) in Base at bitarray.jl:580
convert(::Type{T}, M::AbstractArray) where T<:OffsetArrays.OffsetArray in OffsetArrays at /Users/svenduve/.julia/packages/OffsetArrays/TcCEq/src/OffsetArrays.jl:256
convert(::Type{Union{}}, x) in Base at essentials.jl:213
convert(::Type{T}, obj) where T<:FunctionWrappers.FunctionWrapper in FunctionWrappers at /Users/svenduve/.julia/packages/FunctionWrappers/Q5cBx/src/FunctionWrappers.jl:113
convert(::Type{T}, arg) where T<:VecElement in Base at baseext.jl:19
Possible fix, define
convert(::Type{Union{}}, ::SparseArrays.AbstractSparseMatrixCSC)
Stacktrace:
[1] Sparspak.SpkSparseSolver.SparseSolver{Int64, Float64}(p::SparseArrays.SparseMatrixCSC{Float64, Int64}, slvr::Sparspak.SpkSparseBase._SparseBase{Int64, Float64}, n::Int64, ma::Int64, na::Int64, mc::Int64, nc::Int64, _inmatrixdone::Bool, _orderdone::Bool, _symbolicdone::Bool, _factordone::Bool, _trisolvedone::Bool, _refinedone::Bool, _condestdone::Bool) (repeats 2 times)
@ Sparspak.SpkSparseSolver ~/.julia/packages/Sparspak/5cKXy/src/SparseMethod/SpkSparseSolver.jl:18
[2] Sparspak.SpkSparseSolver.SparseSolver(m::SparseArrays.SparseMatrixCSC{Float64, Int64})
@ Sparspak.SparseCSCInterface ~/.julia/packages/Sparspak/5cKXy/src/SparseCSCInterface/SparseCSCInterface.jl:189
[3] sparspaklu(m::SparseArrays.SparseMatrixCSC{Float64, Int64}; factorize::Bool)
@ Sparspak.SparseCSCInterface ~/.julia/packages/Sparspak/5cKXy/src/SparseCSCInterface/SparseCSCInterface.jl:219
[4] init_cacheval(#unused#::LinearSolve.SparspakFactorization, A::SparseArrays.SparseMatrixCSC{Float64, Int64}, b::Vector{Float64}, u::Vector{Float64}, Pl::SciMLOperators.IdentityOperator{4}, Pr::SciMLOperators.IdentityOperator{4}, maxiters::Int64, abstol::Float64, reltol::Float64, verbose::Bool, assumptions::LinearSolve.OperatorAssumptions{true})
@ LinearSolve ~/.julia/packages/LinearSolve/fBayl/src/factorization.jl:514
[5] init(::SciMLBase.LinearProblem{Nothing, true, SparseArrays.SparseMatrixCSC{Float64, Int64}, Vector{Float64}, SciMLBase.NullParameters, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}, ::LinearSolve.SparspakFactorization; alias_A::Bool, alias_b::Bool, abstol::Float64, reltol::Float64, maxiters::Int64, verbose::Bool, Pl::SciMLOperators.IdentityOperator{4}, Pr::SciMLOperators.IdentityOperator{4}, assumptions::LinearSolve.OperatorAssumptions{true}, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ LinearSolve ~/.julia/packages/LinearSolve/fBayl/src/common.jl:117
[6] init(::SciMLBase.LinearProblem{Nothing, true, SparseArrays.SparseMatrixCSC{Float64, Int64}, Vector{Float64}, SciMLBase.NullParameters, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}, ::LinearSolve.SparspakFactorization)
@ LinearSolve ~/.julia/packages/LinearSolve/fBayl/src/common.jl:88
[7] solve(::SciMLBase.LinearProblem{Nothing, true, SparseArrays.SparseMatrixCSC{Float64, Int64}, Vector{Float64}, SciMLBase.NullParameters, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}, ::LinearSolve.SparspakFactorization; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ LinearSolve ~/.julia/packages/LinearSolve/fBayl/src/common.jl:157
[8] solve(::SciMLBase.LinearProblem{Nothing, true, SparseArrays.SparseMatrixCSC{Float64, Int64}, Vector{Float64}, SciMLBase.NullParameters, Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}}, ::LinearSolve.SparspakFactorization)
@ LinearSolve ~/.julia/packages/LinearSolve/fBayl/src/common.jl:154
[9] macro expansion
@ ~/.julia/packages/LinearSolve/fBayl/src/LinearSolve.jl:102 [inlined]
[10] top-level scope
@ ~/.julia/packages/SnoopPrecompile/1XXT1/src/SnoopPrecompile.jl:62
[11] include
@ ./Base.jl:419 [inlined]
[12] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
@ Base ./loading.jl:1554
[13] top-level scope
@ stdin:1
in expression starting at /Users/svenduve/.julia/packages/LinearSolve/fBayl/src/LinearSolve.jl:1
in expression starting at stdin:1
ERROR: Failed to precompile LinearSolve [7ed4a6bd-45f5-4d41-b270-4a48e9bafcae] to /Users/svenduve/.julia/compiled/v1.8/LinearSolve/jl_FFaxJU.
Stacktrace:
[1] error(s::String)
@ Base ./error.jl:35
[2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
@ Base ./loading.jl:1707
[3] compilecache
@ ./loading.jl:1651 [inlined]
[4] _require(pkg::Base.PkgId)
@ Base ./loading.jl:1337
[5] _require_prelocked(uuidkey::Base.PkgId)
@ Base ./loading.jl:1200
[6] macro expansion
@ ./loading.jl:1180 [inlined]
[7] macro expansion
@ ./lock.jl:223 [inlined]
[8] require(into::Module, mod::Symbol)
@ Base ./loading.jl:1144

`

I am on Julia 1.8.5, just updated.

Many thanks.

[ChrisRackauckas]

Can you share versioninfo()? It looks like it might be the same as https://discourse.julialang.org/t/precompile-error-in-differentialequations-jl/94086/5 and #266, which might only be on Mac M1? So versioninfo() would be helpful because this also looks to be the same, and it looks to be the Homebrew build?

[SvenDuve]

julia> versioninfo() Julia Version 1.8.5 Commit 17cfb8e65e* (2023-01-08 06:45 UTC) Platform Info: OS: macOS (arm64-apple-darwin22.1.0) CPU: 10 × Apple M1 Pro WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1) Threads: 1 on 8 virtual cores

, and yes its a home-brew build.

is M1 an issue?

[ChrisRackauckas]

Looks like it might be. I'll need to track down someone who has one. @YingboMa @chriselrod have you experienced this?

[j-fu]

I found a colleague, he had no problem.

[ChrisRackauckas]

Was it the homebrew version?

[chriselrod]

The error is:

[ Info: Precompiling LinearSolve [7ed4a6bd-45f5-4d41-b270-4a48e9bafcae]
ERROR: LoadError: MethodError: convert(::Type{Union{}}, ::SparseArrays.SparseMatrixCSC{Float64, Int64}) is ambiguous. Candidates:
convert(T::Type{<:LinearAlgebra.Bidiagonal}, m::AbstractMatrix) in LinearAlgebra at /opt/homebrew/Cellar/julia/1.8.5/share/julia/stdlib/v1.8/LinearAlgebra/src/bidiag.jl:203
convert(T::Type{<:SparseArrays.AbstractSparseMatrixCSC}, m::AbstractMatrix) in SparseArrays at /opt/homebrew/Cellar/julia/1.8.5/share/julia/stdlib/v1.8/SparseArrays/src/sparsematrix.jl:745
convert(T::Type{<:LinearAlgebra.SymTridiagonal}, m::SparseArrays.AbstractSparseMatrixCSC) in SparseArrays at /opt/homebrew/Cellar/julia/1.8.5/share/julia/stdlib/v1.8/SparseArrays/src/sparsematrix.jl:749
convert(T::Type{<:LinearAlgebra.LowerTriangular}, m::SparseArrays.AbstractSparseMatrixCSC) in SparseArrays at /opt/homebrew/Cellar/julia/1.8.5/share/julia/stdlib/v1.8/SparseArrays/src/sparsematrix.jl:753
convert(T::Type{<:SparseArrays.SparseVector}, m::SparseArrays.AbstractSparseMatrixCSC) in SparseArrays at /opt/homebrew/Cellar/julia/1.8.5/share/julia/stdlib/v1.8/SparseArrays/src/sparsevector.jl:466
convert(T::Type{<:LinearAlgebra.Tridiagonal}, m::SparseArrays.AbstractSparseMatrixCSC) in SparseArrays at /opt/homebrew/Cellar/julia/1.8.5/share/julia/stdlib/v1.8/SparseArrays/src/sparsematrix.jl:751
convert(T::Type{<:LinearAlgebra.UpperTriangular}, m::SparseArrays.AbstractSparseMatrixCSC) in SparseArrays at /opt/homebrew/Cellar/julia/1.8.5/share/julia/stdlib/v1.8/SparseArrays/src/sparsematrix.jl:755
convert(T::Type{<:LinearAlgebra.Diagonal}, m::SparseArrays.AbstractSparseMatrixCSC) in SparseArrays at /opt/homebrew/Cellar/julia/1.8.5/share/julia/stdlib/v1.8/SparseArrays/src/sparsematrix.jl:747
convert(::Type{Union{}}, a::AbstractArray) in Base at array.jl:618
convert(::Type{SA}, a::AbstractArray) where SA<:StaticArraysCore.StaticArray in StaticArrays at /Users/svenduve/.julia/packages/StaticArrays/jA1zK/src/convert.jl:194
convert(::Type{T}, a::AbstractArray) where T<:Array in Base at array.jl:617
convert(T::Type{<:BitArray}, a::AbstractArray) in Base at bitarray.jl:580
convert(::Type{T}, M::AbstractArray) where T<:OffsetArrays.OffsetArray in OffsetArrays at /Users/svenduve/.julia/packages/OffsetArrays/TcCEq/src/OffsetArrays.jl:256
convert(::Type{Union{}}, x) in Base at essentials.jl:213
convert(::Type{T}, obj) where T<:FunctionWrappers.FunctionWrapper in FunctionWrappers at /Users/svenduve/.julia/packages/FunctionWrappers/Q5cBx/src/FunctionWrappers.jl:113
convert(::Type{T}, arg) where T<:VecElement in Base at baseext.jl:19
Possible fix, define
convert(::Type{Union{}}, ::SparseArrays.AbstractSparseMatrixCSC)

Union{} looks wrong, so I'll have to check where that came from.

I'm always suspicious of Julia installs that aren't official binaries/from juliaup/built from source.
This at least isn't an error complaining about llvm or a random segfault.

Union{} looks like someone used Base.promote_op/Core.compiler.return_type to pick a type to convert to (or container eltype), but the function signature hit something that does not return/is known to error.
Which maybe could be related.
I'll try on my M1 soon.

[ChrisRackauckas]

We've got a dev recreating the error, finally 😅😅😅😅😅😅😅😅😅

[j-fu]

It went well on my colleague's M1 with homebrew 3.6.14.

Yeah Union{} of course looks wrong. It is in an assignement of a struct element declared as

p::Union{Problem{IT, FT},SparseMatrixCSC{FT,IT}}

[chriselrod]

No, I copy/pasted from above to make it readable lol
I don't have a problem with LinearSolve myself...

julia> using LinearSolve
 │ Package LinearSolve not found, but a package named LinearSolve is available from a registry. 
 │ Install package?
 │   (diffeq) pkg> add LinearSolve 
 └ (y/n/o) [y]: 
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
    Updating `~/Documents/progwork/julia/env/diffeq/Project.toml`
  [7ed4a6bd] + LinearSolve v1.35.0
  No Changes to `~/Documents/progwork/julia/env/diffeq/Manifest.toml`

julia> versioninfo()
Julia Version 1.10.0-DEV.501
Commit 2a9f4412a0 (2023-02-05 19:46 UTC)
Platform Info:
  OS: Linux (aarch64-unknown-linux-gnu)
  CPU: 8 × unknown
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, apple-m1)
  Threads: 8 on 8 virtual cores
Environment:
  JULIA_PATH = @.
  JULIA_NUM_THREADS = 8

(diffeq) pkg> st -m LinearSolve
Status `~/Documents/progwork/julia/env/diffeq/Manifest.toml`
  [7ed4a6bd] LinearSolve v1.35.0

[ChrisRackauckas]

...

[chriselrod]

It went well on my colleague's M1 with homebrew 3.6.14.

Yeah Union{} of course looks wrong. It is in an assignement of a struct element declared as

p::Union{Problem{IT, FT},SparseMatrixCSC{FT,IT}}

I can't think of many ways for Union{} to appear, and can't think of any from this. Trying to play with minimal examples, things seem to behave as I'd expect:

julia> struct FooUnion
           x::Union{Float64,Int64}
       end

julia> FooUnion(1f0) # calls convert(::Type{Union{Float64, Int64}}, x::Float32)
ERROR: MethodError: no method matching Union{Float64, Int64}(::Float32)

Closest candidates are:
  (::Type{T})(::T) where T<:Number
   @ Core boot.jl:793
  (::Type{T})(::Base.TwicePrecision) where T<:Number
   @ Base twiceprecision.jl:266
  (::Type{T})(::Complex) where T<:Real
   @ Base complex.jl:44
  ...

Stacktrace:
 [1] convert(#unused#::Type{Union{Float64, Int64}}, x::Float32)
   @ Base ./number.jl:7
 [2] FooUnion(x::Float32)
   @ Main ./REPL[2]:2
 [3] top-level scope
   @ REPL[3]:1

julia> struct FooUnion1
           x::Union{AbstractFloat,Nothing}
``
       end

julia> FooUnion1(1f0)
FooUnion1(1.0f0)

julia> FooUnion1(nothing)
FooUnion1(nothing)

julia> FooUnion1(false) # promotes to float
FooUnion1(0.0)

SparseArrays.SparseMatrixCSC{Float64, Int64}) should have no problem being assigned to a p::Union{Problem{IT, FT},SparseMatrixCSC{FT,IT}} field.

Mind running ] st -m LinearSolve Sparspak?

[j-fu]

Other OP is on https://discourse.julialang.org/t/precompile-error-in-differentialequations-jl/94086/11

[j-fu]

Yeah this already goes through all kinds of CI on win/mac(intel)/linux. So this appears really wild to me.

[jredondoyuste]

Hi! Other person with the issue here: my Homebrew is 3.6.20, that's how I built Julia

[SvenDuve]

I am on Homebrew 3.6.20 FYI.

I'll remove the Homebrew install and try the official binary now.