need better default behavior for `HTTP.Form`

[ExpandingMan]

I've been struggling for a while now to use HTTP.Form to upload files to Slack. Unfrotunately for me, Slack only gives the extremely unhelpful response invalid_form_data. After some digging on the internet, I suspect that the output of HTTP.Form is not quite what Slack is expecting.

See, in particular these examples. One thing I noticed is that, in these examples, both filename and the file itself are contained in the same section, whereas HTTP.Form outputs these as completely separate sections.

If I ever get the upload to Slack working, I will post what I had to do to get it to accept the form.

Also, it doesn't seem to me that HTTP.Form is including the line where it needs to specify the boundary.

[ExpandingMan]

Ok, I've figured out what's going on here and it's not what I thought it was.

First of all, the boundary needs to appear in the header. From a user perspective, this can be extremely mysterious. There is a content_type function that produces the correct header pair for you, but it certainly doesn't look like part of the public API.

The other problem is that this only has the correct section for a file if one uses an IOStream. This isn't helpful if one wants to upload from an in-memory buffer such as IOBuffer.

[ExpandingMan]

Ok, for the record, here is the code I'm using to hack this

function _writemultipartheader(io::IOBuffer, i::IO, filename::AbstractString)
    write(io, "; filename=\"$filename\"\r\n")                                    
    write(io, "Content-Type: $(HTTP.sniff(i))\r\n\r\n")                          
    nothing                                                                                                   
end                                                                
function form(d, filename)                                                                                 
    boundary = string(rand(UInt128), base=16)                                                              
    data = IO[]                 
    io = IOBuffer()                                                                                        
    len = length(d)                                                                                        
    for (i, (k, v)) in enumerate(d)
        write(io, (i == 1 ? "" : "\r\n") * "--" * boundary * "\r\n")
        write(io, "Content-Disposition: form-data; name=\"$k\"")
        if v isa IO                                                                                        
            _writemultipartheader(io, v, filename)                                                         
            seekstart(io)                               
            push!(data, io)                                                                                
            push!(data, v)                              
            io = IOBuffer()                             
        else                                            
            write(io, "\r\n\r\n")
            write(io, v)
            write(io, "\r\n")
        end                            
        i == len && write(io, "--" * boundary * "--" * "\r\n")
    end
    seekstart(io)
    push!(data, io)
    HTTP.Form(data, 1, boundary)
end

I also needed to use content_type(f) on the output of this to get the appropriate header.

The fact that this needs a special header might suggest that we need a specialized PUT method for this type of data.

Note that I added a filename argument here, because if you are uploading something that isn't necessarily a file you need to specify this separately. There was also an extra "\r\n" that was being output which was being interpreted as being part of the file (or at least slack was doing this), so I moved it from where the boundary is written to where the interior data is written.

I'd be happy to make a PR, but what should we change here? Should we add a specialzied HTTP.put method? Is there any way we can implement this without breaking changes? Certainly we also need documentation for this.

[fredrikekre]

So is the problem just that you use HTTP < 0.8.8? From 0.8.8 the header should be set automatically from #473

[ExpandingMan]

Apparently my problem was that I was calling the method put(url, headers, form) whereas it only has the proper behavior for put(url, form). Is there a way to specify additional headers with the latter method? If not, we need a new method.

By the way, apparently the issue with the additional \r\n being interpreted as part of the file only happens for plain text. I'd call this an issue with whatever slack is using to read the data, I put the \r\n back with the boundary.

[fredrikekre]

Hmm, maybe you can just remove the && method == "POST" part from my PR above, that should work. I didn't know if it made sense for other methods than post.

[ExpandingMan]

Oops, sorry, I had meant to say post above, not put. I don't know if it makes sense for put either.

There's still the same issue though: the method you exposed that adds the proper entry for the multipart/form-data header doesn't seem to allow you to add more header entries.

[fredrikekre]

There's still the same issue though: the method you exposed that adds the proper entry for the multipart/form-data header doesn't seem to allow you to add more header entries.

What do you mean? The test I added in that PR sets more headers:

HTTP.jl/test/multipart.jl

Line 2 in 01a5d2e

headers = Dict("User-Agent" => "HTTP.jl")

[fredrikekre]

Doesn't this do what you want @ExpandingMan ?

julia> io = IOBuffer("hello, world");
       data = Dict("file" => HTTP.Multipart("file.txt", io, "text/plain"));
       body = HTTP.Form(data);

julia> print(read(body, String))
--cdd7bc2404756d8121578969150c3c80
Content-Disposition: form-data; name="file"; filename="file.txt"
Content-Type: text/plain

hello, world
--cdd7bc2404756d8121578969150c3c80--

Maybe #676 is enough to close this?

[ExpandingMan]

Yeah I don't remember... I vaguely remember somebody making a change that fixed this. I'll just close this in case I encounter something again that I think is actionable.