Move to Julia 1.0 #113

lucatrv · 2018-09-03T22:59:07Z

No description provided.

lucatrv · 2018-09-06T21:05:10Z

These days I made good progress with CI: initially the tests were failing in less than 1 minute, now they last almost 20 minutes :)

But now I think I'm stuck on the last issue :(

With Julia 1.0, if you run:

using PackageCompiler, Test
compile_package("ArgParse", "SnoopCompile", force = false, reuse = false)

after a while you'll see a series of "Module not found" warnings emitted by the snoop function, and finally an error. I guess the issue may be due to Pkg3, but I'm struggling to fix it, do you have any suggestions?

Notice that I've already changed a few lines of snooping.jl with commit "Fix snooping functions".

NHDaly · 2018-09-06T22:31:35Z

Is that with the current lucatrv:move-to-julia07 branch? Can I pull that down and run the test you describe in order to see it?

lucatrv · 2018-09-06T22:35:05Z

Yes, I'm stuck there

NHDaly · 2018-09-07T12:45:57Z

Hmm, okay, so I tried running it to see what I get. First, i ran into this problem, which I realized was because I didn't have those packages installed:

julia1.0> using PackageCompiler, Test

julia1.0> compile_package("ArgParse", "SnoopCompile", force = false, reuse = false)
┌ Warning: `Pkg.dir(pkgname, paths...)` is deprecated; instead, do `import ArgParse; joinpath(dirname(pathof(ArgParse)), "..", paths...)`.
└ @ Pkg.API /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:454
ERROR: MethodError: no method matching joinpath(::Nothing, ::String)
Closest candidates are:
  joinpath(::String, ::String) at path.jl:217
  joinpath(::AbstractString, ::AbstractString) at path.jl:226
  joinpath(::AbstractString, ::AbstractString, ::AbstractString...) at path.jl:214
Stacktrace:
 [1] (::getfield(PackageCompiler, Symbol("##23#25")))(::Tuple{String,String}) at /Users/daly/.julia/dev/PackageCompiler/src/snooping.jl:104
 [2] map(::getfield(PackageCompiler, Symbol("##23#25")), ::Tuple{Tuple{String,String},Tuple{String,String}}) at ./tuple.jl:163
 [3] snoop_userimg(::String, ::Tuple{String,String}, ::Vararg{Tuple{String,String},N} where N) at /Users/daly/.julia/dev/PackageCompiler/src/snooping.jl:97
 [4] #compile_package#35(::Bool, ::Bool, ::Bool, ::Function, ::Tuple{String,String}, ::Vararg{Tuple{String,String},N} where N) at /Users/daly/.julia/dev/Packa
geCompiler/src/PackageCompiler.jl:106
 [5] (::getfield(PackageCompiler, Symbol("#kw##compile_package")))(::NamedTuple{(:force, :reuse),Tuple{Bool,Bool}}, ::typeof(compile_package), ::Tuple{String,
String}, ::Vararg{Tuple{String,String},N} where N) at ./none:0
 [6] #compile_package#32(::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol,Symbol},NamedTuple{(:force, :reuse),Tuple{Bool,Bool}}}, ::Function, ::String, ::Varar
g{String,N} where N) at /Users/daly/.julia/dev/PackageCompiler/src/PackageCompiler.jl:93
 [7] (::getfield(PackageCompiler, Symbol("#kw##compile_package")))(::NamedTuple{(:force, :reuse),Tuple{Bool,Bool}}, ::typeof(compile_package), ::String, ::Var
arg{String,N} where N) at ./none:0
 [8] top-level scope at none:0

So i installed ArgParse and SnoopCompile.

Now, I get this:

julia1.0> compile_package("ArgParse", "SnoopCompile", force = false, reuse = false)
┌ Warning: `Pkg.dir(pkgname, paths...)` is deprecated; instead, do `import ArgParse; joinpath(dirname(pathof(ArgParse)), "..", paths...)`.
└ @ Pkg.API /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:454
julia_cmd = `/Applications/Julia-1.0.app/Contents/Resources/julia/bin/julia -Cnative -J/Users/daly/.julia/dev/PackageCompiler/sysimg/backup/native/sys.dylib --compile=yes --depwarn=yes --startup-file=no --compiled-modules=no`
ERROR: could not load library "/Users/daly/.julia/dev/PackageCompiler/sysimg/backup/native/sys.dylib"
dlopen(/Users/daly/.julia/dev/PackageCompiler/sysimg/backup/native/sys.dylib, 6): Symbol not found: _jl_inexact_exception
  Referenced from: /Users/daly/.julia/dev/PackageCompiler/sysimg/backup/native/sys.dylib
  Expected in: /Applications/Julia-1.0.app/Contents/Resources/julia/bin/../lib/libjulia.dylib
 in /Users/daly/.julia/dev/PackageCompiler/sysimg/backup/native/sys.dylib
ERROR: IOError: write: broken pipe (EPIPE)
Stacktrace:
 [1] try_yieldto(::typeof(Base.ensure_rescheduled), ::Base.RefValue{Task}) at ./event.jl:196
 [2] wait() at ./event.jl:255
 [3] uv_write(::Base.PipeEndpoint, ::Ptr{UInt8}, ::UInt64) at ./stream.jl:786
 [4] unsafe_write(::Base.PipeEndpoint, ::Ptr{UInt8}, ::UInt64) at ./stream.jl:834
 [5] unsafe_write at ./io.jl:504 [inlined]
 [6] unsafe_write(::Base.PipeEndpoint, ::Base.RefValue{UInt8}, ::Int64) at ./io.jl:502
 [7] write at ./io.jl:505 [inlined]
 [8] write(::Base.PipeEndpoint, ::UInt8) at ./stream.jl:876
 [9] write(::Pipe, ::UInt8) at ./io.jl:224 (repeats 2 times)
 [10] writetag at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Serialization/src/Serialization.jl:136 [inlined]
 [11] writeheader(::Serialization.Serializer{Base.Process}) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Serialization/src/Serialization.jl:656
 [12] serialize(::Base.Process, ::Expr) at /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Serialization/src/Serialization.jl:685
 [13] snoop_vanilla(::String, ::String) at /Users/daly/.julia/dev/PackageCompiler/src/snooping.jl:16
 [14] snoop(::String, ::String, ::String) at /Users/daly/.julia/dev/PackageCompiler/src/snooping.jl:32
 [15] (::getfield(PackageCompiler, Symbol("##23#25")))(::Tuple{String,String}) at /Users/daly/.julia/dev/PackageCompiler/src/snooping.jl:108
 [16] map(::getfield(PackageCompiler, Symbol("##23#25")), ::Tuple{Tuple{String,String},Tuple{String,String}}) at ./tuple.jl:163
 [17] snoop_userimg(::String, ::Tuple{String,String}, ::Vararg{Tuple{String,String},N} where N) at /Users/daly/.julia/dev/PackageCompiler/src/snooping.jl:97
 [18] #compile_package#35(::Bool, ::Bool, ::Bool, ::Function, ::Tuple{String,String}, ::Vararg{Tuple{String,String},N} where N) at /Users/daly/.julia/dev/PackageCompiler/src/PackageCompiler.jl:106
...

What is this backup sys image?: "/Users/daly/.julia/dev/PackageCompiler/sysimg/backup/native/sys.dylib"

lucatrv · 2018-09-08T14:18:15Z

@NHDaly, as for the first issue, yes it is because you need to install ArgParse and SnoopCompile (which anyway are required by PackageCompiler, in future we may want to add an error message once we have fixed other major issues...

As for the second issue, perhaps do you have previous packages and sysimg folders within you /Users/daly/.julia/dev/PackageCompiler/ folder? In this case please remove them before running the test, the image will be rebuilt.

NHDaly · 2018-09-08T16:48:09Z

Hmm yeah I see it too.

julia_cmd = `/Applications/Julia-1.0.app/Contents/Resources/julia/bin/julia -Cnative -J/Users/daly/.julia/dev/PackageCompiler/sysimg/backup/native/sys.dylib --compile=yes --depwarn=yes --startup-file=no --compiled-modules=no`
Launching new julia process to run commands...
Failed to load Revise
ERROR: LoadError: LoadError: ArgumentError: Package Compat not found in current path:
- Run `Pkg.add("Compat")` to install the Compat package.

Stacktrace:
 [1] require(::Module, ::Symbol) at /Users/daly/.julia/dev/PackageCompiler/sysimg/backup/native/sys.dylib:?
 [2] include_relative(::Module, ::String) at /Users/daly/.julia/dev/PackageCompiler/sysimg/backup/native/sys.dylib:?
 [3] include at ./sysimg.jl:29 [inlined]
 [4] include(::String) at /Users/daly/.julia/packages/ArgParse/iJFz/test/runtests.jl:1
 [5] top-level scope at none:0
 [6] include_relative(::Module, ::String) at /Users/daly/.julia/dev/PackageCompiler/sysimg/backup/native/sys.dylib:?
 [7] include(::Module, ::String) at ./sysimg.jl:29
 [8] include(::String) at ./client.jl:388
 [9] top-level scope at /Users/daly/.julia/packages/SnoopCompile/JDKqS/src/SnoopCompile.jl:40
 [10] eval at ./boot.jl:319 [inlined]
 [11] top-level scope at ./none:3
in expression starting at /Users/daly/.julia/packages/ArgParse/iJFz/test/common.jl:2
in expression starting at /Users/daly/.julia/packages/ArgParse/iJFz/test/runtests.jl:3
done.
done.
┌ Warning: Module not found: Compat
└ @ PackageCompiler ~/.julia/dev/PackageCompiler/src/snooping.jl:47

I only have some guesses, but other people would likely have better ideas..

Maybe it's this --compiled-modules=no?
Maybe something's changed with Core.eval, such that you can't actually import modules from within those statements?
https://github.com/JuliaLang/PackageCompiler.jl/pull/113/files#diff-3742cee865c2b0c8839cc675ec593c9cR43

lucatrv · 2018-09-08T20:25:45Z

I tried with --compiled-modules=yes, no luck...

vtjnash · 2018-09-10T19:11:12Z

Why does this package have a private copy of the SnoopCompile code? The bugs in that code have been fixed upstream (some just recently, some in November).

lucatrv · 2018-09-10T20:25:26Z

The code in snooping.jl comes from the original PackageCompiler, before it was merged with StaticJulia. I've just tried to align it with upstream SnoopCompile code (commit "Fix snooping functions") but I guess there are other issues left that I did not notice. @SimonDanisch maybe you can answer this question?

vtjnash · 2018-09-10T20:26:35Z

I started work to update that locally. I'm running into some other issues with the new Pkg / loading code. I'll finish sorting though that and make an update.

NHDaly · 2018-09-10T20:27:59Z

Ah, good catch. thanks Jameson!

vtjnash · 2018-09-10T22:03:00Z

I've got a branch that I think is mostly working: https://github.com/JuliaLang/PackageCompiler.jl/compare/master...vtjnash:lucatrv-move-to-julia07?expand=1

I need to make it better at filtering out test code though, since we're trying to emit stuff like getfield(Main,Symbol("##10#12")` as an argument to map/mapreduce/_all

I'll have to pick up again tomorrow.

lucatrv · 2018-09-11T20:48:15Z

We're closer I think, but I'm still getting some errors unfortunately...

Build shared library "/home/luca/.julia/dev/PackageCompiler/sysimg/backup/native/sys.so":
  `cc -shared -o /home/luca/.julia/dev/PackageCompiler/sysimg/backup/native/sys.so -Wl,--whole-archive /home/luca/.julia/dev/PackageCompiler/sysimg/backup/native/sys.o -Wl,--no-whole-archive -std=gnu99 -I/home/luca/Documents/pkg/julia/julia-1.0.0-linux-x86_64/julia-1.0.0/include/julia -DJULIA_ENABLE_THREADING=1 -fPIC -L/home/luca/Documents/pkg/julia/julia-1.0.0-linux-x86_64/julia-1.0.0/lib -Wl,--export-dynamic -Wl,-rpath,/home/luca/Documents/pkg/julia/julia-1.0.0-linux-x86_64/julia-1.0.0/lib -Wl,-rpath,/home/luca/Documents/pkg/julia/julia-1.0.0-linux-x86_64/julia-1.0.0/lib/julia -ljulia -m64`
Skipping Base.active_repl
Skipping Base.active_repl_backend
Test Summary:        | Pass  Total
Test some Gray stuff |   12     12
Test Summary:           | Pass  Total
bit pattern ambiguities |   17     17
Test Summary:         | Pass  Total
abstract type exports |    7      7
done.
┌ Warning: failed parse of line: 
│   line = "Tuple{typeof(ColorTypes.to_top), Type{ColorTypes.Colorant{T, N}}} where N where T"
└ @ SnoopCompile ~/.julia/packages/SnoopCompile/JDKqS/src/SnoopCompile.jl:122
┌ Warning: failed parse of line: 
│   line = "Tuple{typeof(Base.eltype), Type{C}} where C<:(ColorTypes.Colorant{T, N} where N where T)"
└ @ SnoopCompile ~/.julia/packages/SnoopCompile/JDKqS/src/SnoopCompile.jl:122
┌ Warning: failed parse of line: 
│   line = "Tuple{typeof(ColorTypes.to_top), Type{C}} where C<:(ColorTypes.Colorant{T, N} where N where T)"
└ @ SnoopCompile ~/.julia/packages/SnoopCompile/JDKqS/src/SnoopCompile.jl:122
┌ Warning: failed parse of line: 
│   line = "Tuple{typeof(Base.eltype), Type{ColorTypes.Colorant{T, N}}} where N where T"
└ @ SnoopCompile ~/.julia/packages/SnoopCompile/JDKqS/src/SnoopCompile.jl:122
┌ Warning: `Pkg.dir(pkgname, paths...)` is deprecated; instead, do `import FixedPointNumbers; joinpath(dirname(pathof(FixedPointNumbers)), "..", paths...)`.
└ @ Pkg.API /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:454
Launching new julia process to run commands...
Skipping Base.active_repl
Skipping Base.active_repl_backend
Test Summary: |     Pass     Total
normed        | 63739463  63739463
Test Summary: |   Pass   Total
fixed         | 425852  425852
done.
Julia program file:
  "/home/luca/.julia/dev/PackageCompiler/sysimg/precompile.jl"
Build directory:
  "/home/luca/.julia/dev/PackageCompiler/sysimg"
ERROR: LoadError: UndefVarError: ##17#18 not defined
Stacktrace:
 [1] top-level scope at /home/luca/.julia/dev/PackageCompiler/sysimg/precompile.jl:457
 [2] eval(::Module, ::Any) at ./boot.jl:319
 [3] top-level scope at none:0
 [4] include at ./boot.jl:317 [inlined]
 [5] include_relative(::Module, ::String) at ./loading.jl:1038
 [6] include(::Module, ::String) at ./sysimg.jl:29
 [7] include(::String) at ./client.jl:388
 [8] top-level scope at none:0
in expression starting at /home/luca/.julia/dev/PackageCompiler/sysimg/precompile.jl:2
ERROR: failed process: Process(`/home/luca/Documents/pkg/julia/julia-1.0.0-linux-x86_64/julia-1.0.0/bin/julia -Cnative -J/home/luca/.julia/dev/PackageCompiler/sysimg/backup/native/sys.so --compile=yes --depwarn=yes --startup-file=no --compiled-modules=no --output-o sys.a -e '
  Base.__init__(); Sys.__init__() # initialize "Base" and "Sys" modules
  pushfirst!(Base.DEPOT_PATH, "cache_ji_v1.0.0") # save precompiled modules locally
  include("/home/luca/.julia/dev/PackageCompiler/sysimg/precompile.jl") # include Julia program file'`, ProcessExited(1)) [1]
Stacktrace:
 [1] error(::String, ::Base.Process, ::String, ::Int64, ::String) at ./error.jl:42
 [2] pipeline_error at ./process.jl:695 [inlined]
 [3] #run#505(::Bool, ::Function, ::Cmd) at ./process.jl:653
 [4] run(::Cmd) at ./process.jl:651
 [5] (::getfield(PackageCompiler, Symbol("##4#5")))() at /home/luca/.julia/dev/PackageCompiler/src/static_julia.jl:241
 [6] cd(::getfield(PackageCompiler, Symbol("##4#5")), ::String) at ./file.jl:96
 [7] build_object(::String, ::String, ::String, ::Bool, ::String, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing) at /home/luca/.julia/dev/PackageCompiler/src/static_julia.jl:240
 [8] #static_julia#1(::Nothing, ::Bool, ::Bool, ::String, ::String, ::Nothing, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Bool, ::Nothing, ::Bool, ::Bool, ::String, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::typeof(static_julia), ::String) at /home/luca/.julia/dev/PackageCompiler/src/static_julia.jl:153
 [9] #static_julia at ./none:0 [inlined]
 [10] #build_sysimg#18(::Bool, ::Bool, ::Bool, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::Nothing, ::typeof(PackageCompiler.build_sysimg), ::String, ::String) at /home/luca/.julia/dev/PackageCompiler/src/api.jl:25
 [11] build_sysimg at /home/luca/.julia/dev/PackageCompiler/src/api.jl:24 [inlined]
 [12] #compile_package#35(::Bool, ::Bool, ::Bool, ::Function, ::Tuple{String,String}, ::Vararg{Tuple{String,String},N} where N) at /home/luca/.julia/dev/PackageCompiler/src/PackageCompiler.jl:110
 [13] (::getfield(PackageCompiler, Symbol("#kw##compile_package")))(::NamedTuple{(:force, :reuse),Tuple{Bool,Bool}}, ::typeof(compile_package), ::Tuple{String,String}, ::Vararg{Tuple{String,String},N} where N) at ./none:0
 [14] #compile_package#32(::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol,Symbol},NamedTuple{(:force, :reuse),Tuple{Bool,Bool}}}, ::Function, ::String, ::Vararg{String,N} where N) at /home/luca/.julia/dev/PackageCompiler/src/PackageCompiler.jl:93
 [15] (::getfield(PackageCompiler, Symbol("#kw##compile_package")))(::NamedTuple{(:force, :reuse),Tuple{Bool,Bool}}, ::typeof(compile_package), ::String, ::Vararg{String,N} where N) at ./none:0
 [16] top-level scope at none:0

vtjnash · 2018-09-24T17:35:32Z

Now passing CI at #116

lucatrv added 6 commits September 4, 2018 00:56

Require Julia 0.7

27baad7

Update error messages

d3b40bc

Update @warn messages

b0b84d5

Update @info messages

a88f7ec

Remove legacy code

bc6d7cd

Update hello.jl example

7508a93

lucatrv force-pushed the move-to-julia07 branch from 764642c to 135d6f1 Compare September 3, 2018 23:07

lucatrv added 4 commits September 4, 2018 01:28

Use Serialization

3996c3f

Fix Base.STDOUT deprecation

78db05e

Fix remove_destination deprecation

0986ca1

Fix Base.Test deprecation

38e3643

lucatrv force-pushed the move-to-julia07 branch 2 times, most recently from 3a16478 to 017165e Compare September 3, 2018 23:56

lucatrv changed the title ~~Move to Julia 0.7~~ Move to Julia 1.0 Sep 3, 2018

lucatrv force-pushed the move-to-julia07 branch 5 times, most recently from a6f16ee to cbffc15 Compare September 4, 2018 23:52

lucatrv added 4 commits September 5, 2018 02:11

Update all flags for Julia 1.0

0e0794a

Set 'Travis CI' and 'AppVeyor' to test on Julia 1.0

fa9e9f3

Update using statements

5c9e4b7

Fix replace syntax

292af78

lucatrv force-pushed the move-to-julia07 branch 2 times, most recently from f90f415 to 5d03837 Compare September 5, 2018 08:50

SimonDanisch mentioned this pull request Sep 5, 2018

Can't install package #115

Closed

lucatrv force-pushed the move-to-julia07 branch 3 times, most recently from 8e3e70e to be379cb Compare September 6, 2018 16:37

Fix snooping functions

e4e8b24

Use ArgParse and SnoopCompile for testing

5f2f29a

lucatrv force-pushed the move-to-julia07 branch from be379cb to 5f2f29a Compare September 6, 2018 21:03

vtjnash mentioned this pull request Sep 12, 2018

Move to Julia 1.0 #116

Merged

lucatrv mentioned this pull request Sep 18, 2018

fails Pkg.test with Julia v1.0 #119

Closed

NHDaly mentioned this pull request Sep 18, 2018

Add note to start of the README: 0.7 isn't supported yet #120

Merged

vtjnash closed this Sep 24, 2018

lucatrv deleted the move-to-julia07 branch September 25, 2018 21:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Move to Julia 1.0 #113

Move to Julia 1.0 #113

Uh oh!

lucatrv commented Sep 3, 2018

Uh oh!

lucatrv commented Sep 6, 2018 •

edited

Loading

Uh oh!

NHDaly commented Sep 6, 2018

Uh oh!

lucatrv commented Sep 6, 2018

Uh oh!

NHDaly commented Sep 7, 2018

Uh oh!

lucatrv commented Sep 8, 2018

Uh oh!

NHDaly commented Sep 8, 2018

Uh oh!

lucatrv commented Sep 8, 2018

Uh oh!

vtjnash commented Sep 10, 2018

Uh oh!

lucatrv commented Sep 10, 2018 •

edited

Loading

Uh oh!

vtjnash commented Sep 10, 2018

Uh oh!

NHDaly commented Sep 10, 2018

Uh oh!

vtjnash commented Sep 10, 2018

Uh oh!

lucatrv commented Sep 11, 2018

Uh oh!

vtjnash commented Sep 24, 2018 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Move to Julia 1.0 #113

Move to Julia 1.0 #113

Uh oh!

Conversation

lucatrv commented Sep 3, 2018

Uh oh!

lucatrv commented Sep 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NHDaly commented Sep 6, 2018

Uh oh!

lucatrv commented Sep 6, 2018

Uh oh!

NHDaly commented Sep 7, 2018

Uh oh!

lucatrv commented Sep 8, 2018

Uh oh!

NHDaly commented Sep 8, 2018

Uh oh!

lucatrv commented Sep 8, 2018

Uh oh!

vtjnash commented Sep 10, 2018

Uh oh!

lucatrv commented Sep 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vtjnash commented Sep 10, 2018

Uh oh!

NHDaly commented Sep 10, 2018

Uh oh!

vtjnash commented Sep 10, 2018

Uh oh!

lucatrv commented Sep 11, 2018

Uh oh!

vtjnash commented Sep 24, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

lucatrv commented Sep 6, 2018 •

edited

Loading

lucatrv commented Sep 10, 2018 •

edited

Loading

vtjnash commented Sep 24, 2018 •

edited

Loading