BUG: HDFStore utf8 corrupted reads

[wabu]

When reading back utf8 encoded data, it randomly is corrupted when read back.
The output contains random values or fails with an UnicodeDecodeError.

After commenting out the fast version (astype) in _unconvert_string_array the issue did not show up anymore.

pyton 3.3, numpy 1.8.0, pytables 3.1.0

I'll try to get a reproducible test up in the next days.

[jreback]

are you looking at master? this changed from 0.13.1

this is very tricky because the decoding should work and be very fast via numpy (could be a bug their too). the vectorized decoding is quite slow, FYI (but I bet ALWAYS works)

[wabu]

Yes, its on master. I'll dig deeper into it. Just wanted to already open a ticket for reference.

[jreback]

perfect....note that the current code DOES pass 3.3 / 1.8 on windows / linux,

but if you have a test case that breaks...by all means

[jreback]

@wabu update on this?

[jreback]

@wabu update on this?

[jreback]

@wabu I just merged #6821 pls reopen if you have a test case which fails this. It should automatically decode via a fast loop, and fall back to a vectorized decode otherwise.

[wabu]

sorry for the long absence. It's still a problem, so I created a test that fails. It still is invalid for older python version but shows that there is a problem.

I also tried to reproduce it with plain numpy.astype('U').astype(object), but it didn't show up. Moreover pytables can read the correct data from the hdf5 file.

[jreback]

hmm
can u try this on master
and show_versions

[wabu]

here's a more complete test.
It creates random valid utf8 string, stores them with pandas, reads back with plain pytables and unconverts both with astype and vectorized call, all running fine. But when read back with select, it begins to get wrong data. In productive code, I also did get total wrong strings, here mostly empty strings ...

INSTALLED VERSIONS
------------------
commit: None
python: 3.3.2.final.0
python-bits: 64
OS: Linux
OS-release: 3.2.0-61-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_NZ.UTF-8

pandas: 0.14.0rc1-73-g8793356
nose: 1.3.0
Cython: 0.19.1
numpy: 1.8.1
scipy: 0.12.0
statsmodels: 0.5.0
IPython: 2.0.0-dev
sphinx: None
patsy: 0.2.1
scikits.timeseries: None
dateutil: 2.2
pytz: 2013b
bottleneck: 0.8.0
tables: 3.1.1
numexpr: 2.3.1
matplotlib: 1.4.x
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
bq: None
apiclient: None
rpy2: None
sqlalchemy: None
pymysql: None
psycopg2: 2.5.1 (dt dec pq3 ext)
ERROR: data read corrupted (44/100)
                         orig h5ed
0   '?K>۫\x01cz;\n\x0bq\x16='   ''
1       'e0=\x14\u05ee0\x13F'   ''
2          '\x19]D,f<\n3P]Ֆ֭'   ''
3  '\x7fݘ\x1cF<U\tt6&\x03?\\'   ''
4   'љ\x1d{\x04\x01D/f[\x16q'   ''
ERROR: data read corrupted (52/100)
                      orig h5ed
0        'e\x1e˵{\x17jqҒ3'   ''
1         '͟ռ_μ(SXn5\x1e$'   ''
2  '?ّQ"\x13[\U00078cb4|%'   ''
3        '8ڼC5Y\x1cCV\x13'   ''
4     'h㷜\x03^h&xo\x171MI'   ''
ERROR: data read corrupted (41/100)
                         orig h5ed
0   'u#ШJ\x0fClA\x0c\x13lts`'   ''
1                 'g"v5ː\x16'   ''
2             '#tyV\x1bȹDSxh'   ''
3  '\x06\x10\x10WZb7/<)ڏ\x11'   ''
4     '\x17xښI\x1dݺ_K9\x19V<'   ''
ERROR: data read corrupted (44/100)
                          orig h5ed
0                '2ӣ\x12:ɠ2wH'   ''
1              'ydO}m\x1d˜]2&'   ''
2  '\x179\x17X8^\x07s\x19_=+̀'   ''
3             '?,08/\x08%ւ#}"'   ''
4              'P螩cftfSN\r\nj'   ''