Breakpoints are ignored in tests only

[eott-siz]

(I wrote about this issue here already, however it seems like this has nothing to do with the plugin for VS Code, hence why I am opening an issue here.)

We are having an issue where breakpoints in our code don't work if the code is used in tests, but do work when executed normally. In the following an example that illustrates this problem:

File structure

project_dir
|> deps
  |> build.jl (empty)
|> src
  |> ModuleToTest.jl
  |> PackageToTest.jl
  |> run.jl
|> test
  |> runtests.jl
|> Manifest.toml
|> Project.toml

ModuleToTest.jl

module ModuleToTest

using Debugger

export do_something

function internal_only(x)
    y = x + 1
    @bp
    return y
end

function do_something(x)
    result = internal_only(x)
    @bp
    return result
end

end # module

PackageToTest.jl

module PackageToTest

using Debugger

include("ModuleToTest.jl")
using .ModuleToTest

function main()
    print("I represent long and complicated calculations\n")
    @bp
    print("The magic number is $(ModuleToTest.do_something(41))\n")
end

end # module

run.jl

using PackageToTest

PackageToTest.main()

included_tests.jl

using PackageToTest.ModuleToTest

@testset "test_do_something" begin
    @test ModuleToTest.do_something(41) == 42
end

@testset "test_internal_only" begin
    @test ModuleToTest.internal_only(41) == 42
end

runtests.jl

using Test

@testset "tests_are_working" begin
    @test true
end

include("included_tests.jl")

Project.toml

name = "PackageToTest"
uuid = "2e0f99b1-0d8f-4296-8d0c-1c50a50d04c8"
authors = []
version = "0.1.0"

[deps]
Debugger = "31a5f54b-26ea-5ae9-a837-f05ce5417438"
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

We "install" the example by running the following in a shell:

julia
] activate .
add Test
add Debugger

Then we can exit to the shell and debug the program like so:

julia --project=.
using Debugger
include("src/PackageToTest.jl")
@enter PackageToTest.main()

This drops into the debug shell and following an input of c should continue to the first breakpoint and output:

In main() at projectdir\src\PackageToTest.jl:8
  8  function main()
  9      print("I represent long and complicated calculations\n")
●10      @bp
>11      print("The magic number is $(ModuleToTest.do_something(41))\n")
 12  end

About to run: (Main.PackageToTest.ModuleToTest.do_something)(41)

The two breakpoints in ModuleToTest.jl are also hit and offer the debug shell at the corresponding points. So far, so good. Now let's run the tests:

julia --project=.
using Debugger
@enter include("test/runtests.jl")

The debug shell is opened in the Base package containing the definition of include, but that's not what we're after so we input c and would expect the tests, which call the code in ModuleToTest.jl, to stop at the breakpoints. Instead, the tests are run, the code is executed in doing so, but no further debug shell is opened. The tests conclude and we're dropped back to the julia REPL.

Are we doing something wrong? Is there something the testing framework does that might break the debugger?

[KristofferC]

@enter include("test/runtests.jl")

I don't think that will recurse down and interpret things in the file. Debugger.jl is mostly designed around debugging function calls.

[eott-siz]

I'm not sure I quite understand the difference between the include in PackageToTest.jl and runtests.jl apart from the latter not including ModuleToTest.jl directly (rather via using the module). In both cases the actual code within ModuleToTest.jl is executed at some point and ought to hit the breakpoints, right?

But perhaps I don't need to understand. I'm not married to the exact way how the files and tests are structured as long as I can debug them and they can live in seperate files.

[KristofferC]

You could put the test in a function and debug that function. Debugger.jl does currently not have good support for debugging top-level code (although it should be possible using https://juliadebug.github.io/JuliaInterpreter.jl/stable/internals/#Toplevel-code-and-world-age).

[eott-siz]

Modified included_tests.jl:

using PackageToTest.ModuleToTest

function test_do_something()
    @test ModuleToTest.do_something(41) == 42
end

@testset "test_do_something" begin
    test_do_something()
end

function test_internal_only()
    @test ModuleToTest.internal_only(41) == 42
end

@testset "test_internal_only" begin
    test_internal_only()
end

Now include("test/runtests.jl") in the julia REPL runs the tests just like before, but afterwards a failing test can be run specifically with @enter test_do_something(), which opens the debug shell and will stop at breakpoints in ModuleToTest.jl.

There are still things I don't understand, but this solution unblocks my work, so I consider this issue resolved. Thanks for all the help!