Incorrect parsing of Float64s

[cjprybol]

Hi Shashi,

Does TextParse use a custom Float parser internally instead base Julia's parse(Float64, x)? I ran some tests and found that ~10% of Floats are parsed incorrectly, on average.

Example code

using TextParse, Base.Test
n = 1_000_000
floats = rand(n);
floats_as_string = join(string.(floats), '\n');
# using the Float parser from Base
@test floats == parse.(Float64, split(floats_as_string, '\n'));
a = csvread(IOBuffer(floats_as_string), header_exists=false)[1][1];
@test_broken a == floats
count(a .!= floats)/n

For 3 example runs I got

0.105029
0.104774
0.105412

[shashi]

You're right, there's a custom function that reads Float64s... It doesn't take care of rounding correctly. We need to call something like https://github.com/JuliaLang/julia/blob/1255e8219101da0175bb262b97ca849c09406061/src/support/strtod.c#L100

It would be nice to stick to the fast path as much as possible though.

[shashi]

interesting read: http://www.exploringbinary.com/how-strtod-works-and-sometimes-doesnt/

also can be a pure-Julia implementation.

[cjprybol]

Thanks for the links! Both are helpful, and the second, very informative.

also can be a pure-Julia implementation.

Julia Summer of Code 2018?

[shashi]

Someone usefully wrote a better implementation here JuliaLang/julia#24015

[cjprybol]

That looks great! I was trying out the strtod function you linked to with examples like ccall((:jl_strtod_c), Float64, (Ptr{UInt8},), "3.14"), but when parsing unexpected values it segfaults. I presume a pure Julia implementation will allow for similar speed but more graceful handling of unexpected inputs.

[quinnj]

FWIW, I'm going to be digging in to fixing the CSV.jl implementation this week; it is also based on the spirit qi implementation, so I'll take a look at the PR to Base that was posted to see if it ends up being more accurate than the one in CSV.jl.

Should we try to make a Float parsing routing generally available to all our packages? I'm worried we wouldn't be able to come to a coherent design, but might be worth a shot. As I've looked into replacing the Base parsing functions, it's just hard to know what and how much information is ever needed; there's the tradeoff of performance and better error messages.

Anyway, hopefully I'll get all the parsing issues for CSV ironed out this week and see if it can't be made more general.

[shashi]

Should we try to make a Float parsing routing generally available to all our packages?

That will be nice. Why can't we get a coherent design?

At some point I'd like to discuss with you how we could move parsing stuff piece by piece into Base. So TextParse and CSV.jl could just be CSV-handling bits. Or better yet, moved to Base.DelimitedFiles if it's not too much extra code.

[davidanthoff]

Fixed almost two years ago :)