Closed
Description
I am trying to read a file 57MB with pandas.csv_read. The file contains a header (5 rows), afterwads integer values and at the end float values:
info
2681087 53329 1287215 1287215 53328
RSA 53328 53328 1287215 0
(I14) (I14) (d25.15) (d25.15)
F 1 5332
1
33
61
92
128
...
165
205
239
272
0.112474585277959E+09
0.126110931411177E+09
0.515995872032845E+09
0.126110931411175E+09
-0.194634413074014E+09
0.112474585277950E+09
...
When I read the txt file:
import pandas as pd
pd.read_csv(file, skiprows=5+n_int_values, header=None, engine='c',
dtype=np.float, low_memory=False)
The result is an error:
---------------------------------------------------------------------------
CParserError Traceback (most recent call last)
<ipython-input-118-699921ac7a12> in <module>()
----> 1 a=pd.read_csv(loc, skiprows=5+n_coloums+n_rows, header=None, engine='c',
low_memory=False, error_bad_lines=False)
C:\Anaconda\lib\site-packages\pandas\io\parsers.pyc in parser_f(filepath_or_buffer, sep, dialect,
compression, doublequote, escapechar, quotechar, quoting, skipinitialspace, lineterminator,
header, index_col, names, prefix, skiprows, skipfooter, skip_footer, na_values, na_fvalues,
true_values, false_values, delimiter, converters, dtype, usecols, engine, delim_whitespace,
as_recarray, na_filter, compact_ints, use_unsigned, low_memory, buffer_lines, warn_bad_lines,
error_bad_lines, keep_default_na, thousands, comment, decimal, parse_dates, keep_date_col,
dayfirst, date_parser, memory_map, float_precision, nrows, iterator, chunksize, verbose,
encoding, squeeze, mangle_dupe_cols, tupleize_cols, infer_datetime_format, skip_blank_lines)
468 skip_blank_lines=skip_blank_lines)
469
--> 470 return _read(filepath_or_buffer, kwds)
471
472 parser_f.__name__ = name
C:\Anaconda\lib\site-packages\pandas\io\parsers.pyc in _read(filepath_or_buffer, kwds)
254 return parser
255
--> 256 return parser.read()
257
258 _parser_defaults = {
C:\Anaconda\lib\site-packages\pandas\io\parsers.pyc in read(self, nrows)
713 raise ValueError('skip_footer not supported for iteration')
714
--> 715 ret = self._engine.read(nrows)
716
717 if self.options.get('as_recarray'):
C:\Anaconda\lib\site-packages\pandas\io\parsers.pyc in read(self, nrows)
1162
1163 try:
-> 1164 data = self._reader.read(nrows)
1165 except StopIteration:
1166 if nrows is None:
pandas\parser.pyx in pandas.parser.TextReader.read (pandas\parser.c:7426)()
pandas\parser.pyx in pandas.parser.TextReader._read_rows (pandas\parser.c:8377)()
pandas\parser.pyx in pandas.parser.raise_parser_error (pandas\parser.c:20728)()
CParserError: Error tokenizing data. C error: Buffer overflow caught - possible malformed input
file.
This happens on pandas 0.16.0, on anaconda python 2.7.8. On an older version - 0.14.1. it works correctly.
Note: When I use engine='python', the txt file is loaded normaly.