Simple model apparently runs out of memory

[sh2pro]

Please see JuliaLang/julia#48418

The machine is an ml.m5.large AWS instance with 8GB of memory.
Should be more than enough for a model with a couple of dozens equations

[ChrisRackauckas]

Closing since this is almost certainly a Base issue with parallel precompilation.

[sh2pro]

can you explain? what is the workaround?

[ChrisRackauckas]

Open Julia with less threads and see if that fixes it.

[sh2pro]

It doesn't

(base) jovyan@julia-ml-m5-large:~/work$ julia --threads=1 test.jl
  Activating project at `~/work`
after using
here we go
Killed
(base) jovyan@julia-ml-m5-large:~/work$

[baggepinnen]

I had problems about a week ago where all CI runs using MTK failed to precompile due to running out of memory. It has gone away by now, so maybe it will for you as well if you update all packages?

[sh2pro]

Seems that ODAEProblem() is running out of memory, so I don't think it's related to precompilation

[ChrisRackauckas]

Sounds like it's a core dump related to codegen then. Can you share the core dump? Write it to a file before the terminal crashes?

[ChrisRackauckas]

What's the reproducible example? I ran the code and get that Segment is not defined at @named s = S(), and indeed Segment is not defined in this example and not in mtkstdlib.

[sh2pro]

Oops. Sorry.

(base) jovyan@julia-ml-t3-medium:~/work$ cat test.jl

using Pkg; Pkg.activate(".")
using ModelingToolkit, ModelingToolkitStandardLibrary
using DifferentialEquations
using ModelingToolkitStandardLibrary.Electrical
using ModelingToolkitStandardLibrary.Blocks

println("after using")
flush(stdout)

function Segment(; name)
    @named R = Resistor(; R=1)
    @named r = Resistor(; R=1)
    @named C = Capacitor(; C=1)
    
    @named p1 = Pin() # top-left
    @named p2 = Pin() # top-right
    @named n = Pin() # bottom
    
    eqs = [
        connect(p1, R.p)
        connect(R.n, p2, r.p)
        connect(r.n, C.p)
        connect(C.n, n)
        ]
    
    return ODESystem(
                eqs, t, [], []; 
                name=name,
                systems=[r, R, C, n, p1, p2])
end

function Strip(;name)
    num_segments = 10
    # construct `num_segments` segments
    segments = [
            Segment(;name=Symbol(:St_, seg))
            for seg in 1:num_segments
        ]

    @named p1 = Pin() # top-left
    @named p2 = Pin() # top-right
    @named n = Pin() # bottom

    eqs = [
        connect(p1, segments[1].p1)
        connect(p2, segments[end].p2)
        [connect(n, seg.n) for seg in segments]...
        [connect(segments[i].p2, segments[i+1].p1) for i in 1:num_segments-1]...
    ]

    return ODESystem(eqs, t, [], [] ; name, 
                systems=[p1, p2, n, segments...])
end
        

@variables t
@named source = Voltage()
@named c = Constant(k=0.01)

@named ground = Ground()
@named strip = Strip()

rc_eqs = [
        connect(c.output, source.V)
        connect(source.p, strip.p1, strip.p2)
        connect(strip.n, source.n, ground.g)
        ]

@named rc_model = ODESystem(rc_eqs, t, systems=[strip, c, source, ground])
sys = structural_simplify(rc_model)

println("here we go")
prob = ODAEProblem(sys, Pair[], (0, 10))
println("still alive")

(base) jovyan@julia-ml-t3-medium:~/work$ julia --threads=1 test.jl
  Activating project at `~/work`
after using
here we go
Killed
(base) jovyan@julia-ml-t3-medium:~/work$ 

[ChrisRackauckas]

@YingboMa able to recreate at all?

[sh2pro]

FYI, here's the environment:

(base) jovyan@julia-ml-t3-medium:~/work$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.5 (2023-01-08)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.8) pkg> activate .
  Activating project at `~/work`

(work) pkg> st
Status `~/work/Project.toml`
  [1b08a953] Dash v1.1.2
  [0c46a032] DifferentialEquations v7.6.0
  [c601a237] Interact v0.10.5
  [2fda8390] LsqFit v0.13.0
  [961ee093] ModelingToolkit v8.43.0
  [16a59e39] ModelingToolkitStandardLibrary v1.10.0
  [f0f68f2c] PlotlyJS v0.18.10
  [91a5bcdd] Plots v1.38.3
  [1986cc42] Unitful v1.12.2

julia> versioninfo()
Julia Version 1.8.5
Commit 17cfb8e65ea (2023-01-08 06:45 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 2 × Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, skylake-avx512)
  Threads: 1 on 2 virtual cores
Environment:
  JULIA_PKGDIR = /opt/julia
  JULIA_VERSION = 1.8.5
  JULIA_DEPOT_PATH = /opt/julia

On a Mac (Apple Silicon), I could see the Julia process allocate more than 50GB before I killed the process (didn't wait for the machine to crash)

[ChrisRackauckas]

Is this isolated then to SciML/OrdinaryDiffEq.jl#1857 and this should be closed? It would make sense if this was the RecursiveFactorization issue mentioned there. I don't see how ModelingToolkit could be the cause, but something weird in LoopVectorization/SIMD codegen at least sounds plausible.

[sh2pro]

@ChrisRackauckas, unfortunately, even after starting from scratch, the same issue remains:

(base) jovyan@julia-ml-m5-large:~/work$ julia test.jl 
  Activating project at `~/work`
after using
here we go
Killed
(base) jovyan@julia-ml-m5-large:~/work$ julia

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.5 (2023-01-08)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.8) pkg> activate .
  Activating project at `~/work`

(work) pkg> st
Status `~/work/Project.toml`
  [0c46a032] DifferentialEquations v7.6.0
  [961ee093] ModelingToolkit v8.43.0
  [16a59e39] ModelingToolkitStandardLibrary v1.10.0
  [f0f68f2c] PlotlyJS v0.18.10
  [91a5bcdd] Plots v1.38.3
  [1986cc42] Unitful v1.12.2

julia> versioninfo()
Julia Version 1.8.5
Commit 17cfb8e65ea (2023-01-08 06:45 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 2 × Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, skylake-avx512)
  Threads: 1 on 2 virtual cores
Environment:
  JULIA_PKGDIR = /opt/julia
  JULIA_VERSION = 1.8.5
  JULIA_DEPOT_PATH = /opt/julia

julia>