tar in import_docker_image fails on Windows

[visr]

Running BinaryBuilder master on Julia 1.3 RC5, I get a Failed to open '\\.\tape0 in import_docker_image.

julia> using BinaryBuilder; BinaryBuilder.runshell(Linux(:x86_64, libc=:glibc), verbose=true)
[ Info: Precompiling BinaryBuilder [12aac903-9f7c-5d81-afc2-d9565ea332ae]
[ Info: Checking to see if c:\Users\visser_mn\.julia\dev\BinaryBuilder\ is encrypted...
[ Info: Couldn't open /proc/mounts, returning...
[ Info: Checking to see if c:\Users\visser_mn\.julia\dev\BinaryBuilder\deps\ is encrypted...
[ Info: Couldn't open /proc/mounts, returning...
[ Info: Importing docker base image from C:\Users\visser_mn\.julia\artifacts\0f1867e8dbf4dfeaa77bb52ad707b86f74f0c80e to julia_binarybuilder_rootfs:v2019.11.22-f0c80e
tar: Failed to open '\\.\tape0'
ERROR: failed process: Process(`tar -c -C 'C:\Users\visser_mn\.julia\artifacts\0f1867e8dbf4dfeaa77bb52ad707b86f74f0c80e' .`, ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error(::Base.ProcessChain) at .\process.jl:538
 [2] #run#563(::Bool, ::typeof(run), ::Base.CmdRedirect) at .\process.jl:440
 [3] run at .\process.jl:438 [inlined]
 [4] #import_docker_image#274(::Bool, ::typeof(BinaryBuilder.import_docker_image), ::BinaryBuilder.CompilerShard, ::String) at c:\Users\visser_mn\.julia\dev\BinaryBuilder\src\DockerRunner.jl:55
 [5] #import_docker_image at .\array.jl:0 [inlined]
 [6] #DockerRunner#275(::String, ::Linux, ::Array{Any,1}, ::Dict{String,String}, ::Bool, ::String, ::String, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Type{BinaryBuilder.DockerRunner}, ::String) at c:\Users\visser_mn\.julia\dev\BinaryBuilder\src\DockerRunner.jl:104
 [7] Type at .\none:0 [inlined]
 [8] #runshell#212(::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(BinaryBuilder.runshell), ::Type{BinaryBuilder.DockerRunner}, ::Linux) at c:\Users\visser_mn\.julia\dev\BinaryBuilder\src\Runner.jl:617
 [9] (::BinaryBuilder.var"#kw##runshell")(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(BinaryBuilder.runshell), ::Type{BinaryBuilder.DockerRunner}, ::Linux) at .\none:0
 [10] #runshell#210(::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:verbose,),Tuple{Bool}}}, ::typeof(BinaryBuilder.runshell), ::Linux) at c:\Users\visser_mn\.julia\dev\BinaryBuilder\src\Runner.jl:609
 [11] (::BinaryBuilder.var"#kw##runshell")(::NamedTuple{(:verbose,),Tuple{Bool}}, ::typeof(BinaryBuilder.runshell), ::Linux) at .\none:0
 [12] top-level scope at REPL[6]:1

Earlier I tried to work around this by using wsl tar or Tar.jl instead, but then I bumped into these issues: #441 (comment).

I guess this is using this tar now:

shell> where tar
C:\Windows\System32\tar.exe
C:\bin\git\usr\bin\tar.exe

Should we use the bundled 7zip here instead?

[staticfloat]

Did your Tar.jl branch work? If so, I'm keen to try using that.

[visr]

I think the branch did its work, although directly afterward it hit into the docker_entrypoint.sh issue, same as for wsl tar.

Tar.jl could work given a few points:

• prevent UndefVarError for Cmode_t on loading package on Windows JuliaIO/Tar.jl#1, fix create to IO JuliaIO/Tar.jl#2, write possibly emptied link to standard header JuliaIO/Tar.jl#3 are merged
• right now it is set to julia 1.4+, not sure why, I see only a readdir(sys_path, sort = false) that I think we can work around
• it is not registered yet, though see also the discussion in new stdlib: Tar.jl JuliaLang/julia#33797 about vendoring it in Pkg

[StefanKarpinski]

Regarding master...visr:tar, I think you can just do this:

io = IOBuffer()
Tar.create(rootfs_path, io)
run(pipeline(
    `docker import - -c $(dockerfile_cmds) $(docker_image(rootfs))`;
    stdin=io, stdout=devnull))
close(io)

The tar command only needs the -C $rootfs_path . stuff because of how you tell the command what to include in the tarball: the Tar.create function always creates an archive that is relative to root that you give it, no need to cd in order to get that effect.

@visr, thanks for trying out Tar and providing fixes!

[visr]

@staticfloat what shall we do here? Tar.jl will likely end up inside Pkg according to JuliaIO/Tar.jl#7 (comment).

I guess for Windows the best alternative for now would be to use the bundled 7z in joinpath(Sys.BINDIR, Base.LIBEXECDIR, "7z.exe")? Would be happy to make a PR for that.

[StefanKarpinski]

I've release Tar.jl v0.1.0 now, so you can go ahead and use it as a dependency. We'll vendor a copy inside of Pkg, but you can just use it as a normal dependency instead.

[fingolfin]

Is this issue still ongoing?

[visr]

I think so, since system tar is still used:

https://github.com/JuliaPackaging/BinaryBuilderBase.jl/blob/c67ccbebfa755426a34ea31aba7e4a2747c3dae2/src/DockerRunner.jl#L56-L59

This PR would switch that over to Tar.jl, but see this comment about possible performance issues:
#548 (comment)