Skip to content

Adds missing __next__ method to SAS7BDATReader needed for iteration #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

spillz
Copy link
Owner

@spillz spillz commented Aug 6, 2016

@benjello
Copy link

Is this PR working. If the answer is positive, it would be nice to have it in upstream.

@benjello
Copy link

I insert the next method you provided but I ended up the following error

Exception ValueError: ValueError('Unexpected non-zero end_of_first_byte',) in 'pandas.io.sas.saslib.Parser.process_byte_array_with_data' ignored
Exception ValueError: ValueError('Unexpected non-zero end_of_first_byte',) in 'pandas.io.sas.saslib.Parser.process_byte_array_with_data' ignored
Exception ValueError: ValueError('Unexpected non-zero end_of_first_byte',) in 'pandas.io.sas.saslib.Parser.process_byte_array_with_data' ignored
Exception ValueError: ValueError('Unexpected non-zero end_of_first_byte',) in 'pandas.io.sas.saslib.Parser.process_byte_array_with_data' ignored
...
Exception ValueError: ValueError('Unexpected non-zero end_of_first_byte',) in 'pandas.io.sas.saslib.Parser.process_byte_array_with_data' ignored
Exception ValueError: ValueError('Unexpected non-zero end_of_first_byte',) in 'pandas.io.sas.saslib.Parser.process_byte_array_with_data' ignored
Exception ValueError: ValueError('Unexpected non-zero end_of_first_byte',) in 'pandas.io.sas.saslib.Parser.process_byte_array_with_data' ignored
Exception ValueError: ValueError('Unexpected non-zero end_of_first_byte',) in 'pandas.io.sas.saslib.Parser.process_byte_array_with_data' ignored
Traceback (most recent call last):
  File "./fonction_publique/scripts/clean_raw_career.py", line 43, in <module>
    sys.exit(main())
  File "./fonction_publique/scripts/clean_raw_career.py", line 38, in main
    chunksize = chunksize,
  File "/home/benjello/openfisca/fonction-publique/fonction_publique/raw_data_cleaner.py", line 202, in main
    chunksize = chunksize,
  File "/home/benjello/openfisca/fonction-publique/fonction_publique/raw_data_cleaner.py", line 117, in format_generation
    generation = get_subset('generation', file_path, debug = debug, chunksize = chunksize)
  File "/home/benjello/openfisca/fonction-publique/fonction_publique/base.py", line 122, in wrap
    ret = f(*args, **kwargs)
  File "/home/benjello/openfisca/fonction-publique/fonction_publique/raw_data_cleaner.py", line 36, in get_subset
    for chunk in reader:
  File "/home/benjello/.local/lib/python2.7/site-packages/pandas/io/common.py", line 113, in <lambda>
    BaseIterator.next = lambda self: self.__next__()
  File "/home/benjello/.local/lib/python2.7/site-packages/pandas/io/sas/sas7bdat.py", line 585, in __next__
    return self.read(nrows=self.chunksize or 1)
  File "/home/benjello/.local/lib/python2.7/site-packages/pandas/io/sas/sas7bdat.py", line 607, in read
    rslt = self._chunk_to_dataframe()
  File "/home/benjello/.local/lib/python2.7/site-packages/pandas/io/sas/sas7bdat.py", line 649, in _chunk_to_dataframe
    dtype=self.byte_order + 'd')
  File "/home/benjello/.local/lib/python2.7/site-packages/pandas/core/frame.py", line 2417, in __setitem__
    self._set_item(key, value)
  File "/home/benjello/.local/lib/python2.7/site-packages/pandas/core/frame.py", line 2483, in _set_item
    value = self._sanitize_column(key, value)
  File "/home/benjello/.local/lib/python2.7/site-packages/pandas/core/frame.py", line 2638, in _sanitize_column
    value = _sanitize_index(value, self.index, copy=False)
  File "/home/benjello/.local/lib/python2.7/site-packages/pandas/core/series.py", line 2798, in _sanitize_index
    raise ValueError('Length of values does not match length of ' 'index')
ValueError: Length of values does not match length of index

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Issue iterating with pd.read_sas
2 participants