read_csv() & EOF character in string cause parsing issue

[stephenjshaw]

While importing large text files using read_csv we occasionally get an EOF (End of File ) character within a string, which causes an exception: "Error tokenizing data. C error: EOF inside string starting at line. 844863" . This occurs even with "error_bad_lines = False"..

Further, the line stated in the error message is not the line containing the EOF character. In this particular case the actual row was approx. 230 rows before the one stated, which hinders exception handling. (I now see this difference was caused by other "bad_lines" that were being skipped - the quoted error line is correct but the imported rows was less.)

I feel it would be appropriate if "error_bad_lines = False" handled this exception and allowed such rows to be skipped.

I note that when importing this text file into Excel, the "premature" EOF is simply ignored.

We are running on Windows 8 , with python version 2.7 and pandas version 0.12

[stephenjshaw]

Further investigation using a hex editor has revealed what is going on:

• I added 0x1A ("EOF") to a different file and it did not cause any problems. Pandas read_csv imported it without error.
• I parsed every line of the problematic CSV individually, until I isolated the one causing the problem. It was over 3000 rows after the stated row number in the error message.
• the row in question had a column with a double quote mark following the delimiter - there were not supposed to be any quote marks in the file. There was no second double quote in the column, or on the row
• I think the quote mark caused the import to look for a second terminating double quote, ignoring column delimiters and end of line markers until it reached the end of the file. When it didn't find one before the end of the file, Im speculating that triggered the "EOF inside a string" error message.
• I was able to work around the problem by setting the quotechar to be the same as the delimiter, while tells read_csv to ignore all quotes. It now imports the file perfectly.
• I still think "error_bad_lines" should catch this by checking if any row contains a column with a missing terminating quote.
• one reason I think this is important is that by adding a second such double quote, many lines apart, I was able to "fool" the system into skipping all the intervening lines, even though only two rows had an error.

[guyrt]

@stephenjshaw are you able to try this on the master branch?

[rcompton]

On pandas 0.13.1, I had the exact same problem and solution.

[DataJunkie]

I am having the same issue and cannot find any offending characters in the lines near the line number given. Is there some way to search for weird characters given I have no clue where the issue is?

[hayd]

another user seeing this/similar: http://stackoverflow.com/q/24005761/1240268

[rcompton]

Note that, according to the documentation at http://pandas.pydata.org/pandas-docs/version/0.13.1/generated/pandas.io.parsers.read_csv.html , error_bad_lines=False only means that lines with too many fields will be skipped. If there's other problems with your data, .read_csv() will fail rather than skip the problematic lines, cf #6478 and https://stackoverflow.com/questions/22026181/pandas-warn-bad-lines-false-and-error-bad-lines-false-is-still-trying-to-parse-b

[jreback]

I don't think this is that hard to fix (essentially the low-level reader returns on EOF, but simple enough to check if that's actually the end of the file by reading again, if not, then can just ignore I think / remove that line).

anyone have a couple of test cases (e.g. need EOF inside a quote and outside). can generate the, but maybe @stephenjshaw already has a bit of code to do this?

[yyl]

Met the same problem here and solved it by @stephenjshaw solution.