Skip to content

Datetime resolution coercion #10249

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 8, 2015
Merged

Datetime resolution coercion #10249

merged 1 commit into from
Jun 8, 2015

Conversation

cmeeren
Copy link
Contributor

@cmeeren cmeeren commented Jun 2, 2015

Fixes #10245

Without this fix, if the date_parse function passed to e.g. read_csv outputs a np.datetime64 array, then it must be of ns resolution. With this fix, all time resolutions may be used.

@cmeeren cmeeren changed the title Datetime res coercion Datetime resolution coercion Jun 2, 2015
parse_dates={'datetime': ['week', 'sow']},
index_col=['datetime', 'prn'])

self.assertEqual(df.index[0][0], datetime(2013, 11, 3, 19))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

construct an actual frame and use assert_frame_equal

@jreback jreback added Bug IO CSV read_csv, to_csv Datetime Datetime data dtype labels Jun 2, 2015
@cmeeren
Copy link
Contributor Author

cmeeren commented Jun 4, 2015

For some reason 706b8df fails Travis on my fork. I can't figure out why. The error doesn't seem to be related to my testing function (which was the only thing I changed, apart from importing MultiIndex at the top).

@jreback
Copy link
Contributor

jreback commented Jun 4, 2015

pull from master again and rebase. i fixed the issue.

@cmeeren
Copy link
Contributor Author

cmeeren commented Jun 4, 2015

Still failed Travis in my fork

@jreback
Copy link
Contributor

jreback commented Jun 4, 2015

oh, that was a generic message I sent out. You might have a specific issue then.

@cmeeren
Copy link
Contributor Author

cmeeren commented Jun 4, 2015

Then I really don't understand what's going on.

  • 1549094 passes (after fixing the issue and making the test)
  • 706b8df fails 1/5 (after I updated my test) fails in a test I haven't touched
  • 3c348dd fails 4/5 (after rebasing) fails with another error

I'm at a loss here.

new_col = parser(*to_parse)
if hasattr(new_col, 'dtype') and any(r in str(new_col.dtype) for r in
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is all this for ? this should not be necessary

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What? Oh, man. That was the first fix I came up with, scrubbed in favor of your suggestion. No idea how it made its way in.

the two first commits in the PR, b6d07b4 and 7d6f7c4, should be reverted. How do you propose I do this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rebase on master, and just delete them, see http://pandas-docs.github.io/pandas-docs-travis/contributing.html#getting-started-with-git

you basically

git rebase -i master

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I see what's wrong. I had previously committed the two unwanted commits to my fork's master branch (before pulling the updated master branch). I tried git revert but that caused merge conflicts so I reset. How can we solve this with a minimum of mess? Should we just close this PR and I'll open a new one with the good commits?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you just rebase it and delete the commits you don't want.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know how to "rebase" the master branch. I tried git rebase -i on master, and it just said noop where the commit list should be.

@jreback
Copy link
Contributor

jreback commented Jun 4, 2015

try git rebase -i origin/master
and see https://git-scm.com/book/en/v2/Git-Branching-Rebasing

@cmeeren
Copy link
Contributor Author

cmeeren commented Jun 4, 2015

There we are. Ended up adding pydata/pandas as a remote and rebasing this PR branch on top of that. Diff seems correct now.

@cmeeren
Copy link
Contributor Author

cmeeren commented Jun 6, 2015

Build passed, will this be merged? Or is anything else needed?

@shoyer shoyer added this to the 0.16.2 milestone Jun 7, 2015
@shoyer
Copy link
Member

shoyer commented Jun 7, 2015

@cmeeren thanks for the nudge. I think this looks pretty much good now. All we need now is a release note in what's new in 0.16.2.txt. You can put this under bug fixes.

Once you've done that, please squash to one commit.

@cmeeren
Copy link
Contributor Author

cmeeren commented Jun 7, 2015

Oh no, again the thing with the wrong commits being included on rebase. I'll rebase onto upstream/master and try again.

@cmeeren
Copy link
Contributor Author

cmeeren commented Jun 7, 2015

There we are.

@@ -152,3 +152,4 @@ Bug Fixes

- Bug to handle masking empty ``DataFrame``(:issue:`10126`)
- Bug where MySQL interface could not handle numeric table/column names (:issue:`10255`)
- Bug where ``read_csv`` and similar failed if making ``MultiIndex`` and ``date_parser`` returned ``datetime64`` array of other time resolution than ``[ns]``.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add the issue number at the end

jreback added a commit that referenced this pull request Jun 8, 2015
@jreback jreback merged commit 48165c9 into pandas-dev:master Jun 8, 2015
@jreback
Copy link
Contributor

jreback commented Jun 8, 2015

thanks!

@cmeeren
Copy link
Contributor Author

cmeeren commented Jun 8, 2015

Oh, I thought you wanted the issue number in the "whatsnew" document. I hadn't pushed that yet. But you're the boss. :)

@jreback
Copy link
Contributor

jreback commented Jun 8, 2015

I forgot I asked you, and just did it. 0a428e7

thanks!

@cmeeren
Copy link
Contributor Author

cmeeren commented Jun 8, 2015

👍

@cmeeren cmeeren deleted the datetime_res_coercion branch June 8, 2015 13:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Datetime Datetime data dtype IO CSV read_csv, to_csv
Projects
None yet
Development

Successfully merging this pull request may close these issues.

numpy error using read_csv with parse_dates=[...] and index_col=[...]
3 participants