BUG: DateOffset weekday around DST produces unexpected results

[adamgreenhall]

I want next Monday:

def next_week_start(t):
    return t + pd.DateOffset(weekday=0)

But for DST (fall back is Sunday 2013-11-03):

 >> next_week_start(pd.Timestamp("2013-10-30", tz="US/Pacific"))
 Timestamp('2013-11-03 23:00:00-0800', tz='US/Pacific')

which is Sunday at 11pm, not Monday.

[jreback]

cc @rockg

can you take a look?

[adamgreenhall]

Testing:

def next_monday(t):
    return t + pd.DateOffset(weekday=0)

tz = "US/Pacific"
def test_week_start():    
    # a Tuesday - should return next week
    t = next_monday(pd.Timestamp("2013-01-01", tz=tz))
    assert(t.hour == 0 and t.dayofweek == 0 and t.day == 7)

    # a Monday - should return same day
    t = next_monday(pd.Timestamp("2013-01-07", tz=tz))
    assert(t.hour == 0 and t.dayofweek == 0 and t.day == 7)

     # fall back is Sun. Nov 03
    t = next_monday(pd.Timestamp("2013-10-30", tz=tz))
    assert(t.hour == 0 and t.dayofweek == 0)

    # spring fwd is Sun. Mar 10
    t = next_monday(pd.Timestamp("2013-03-08", tz=tz))
    assert(t.hour == 0  and t.dayofweek == 0)

[jreback]

see the last part of this section: http://pandas.pydata.org/pandas-docs/dev/timeseries.html#timeseries-timezone

there is a new kw, infer_dst that you can set in localize to 'fix' this problem; might need to be enabled here somehow

[rockg]

Also, there is a bug in dateutil (the default for DateOffset) which does not take into account tz when going from a dst to non-dst time. I don't know if we want to fix it there or account for it in DateOffset (the easiest fix is to remove the tzinfo, apply the offset, and then reapply the tzinfo).

[rockg]

@adamgreenhall @jreback This is what I'm talking about. The issue is that the tzinfo remains the same offset despite the dst offset actually changing. So tzinfo needs to be removed, the offset applied, and then the result needs to be localized using the initial tzinfo. This should probably be implemented in DateOffset for everything.

>>> ts = pd.Timestamp('2013-03-08', tz='US/Pacific')
>>> ts
Timestamp('2013-03-08 00:00:00-0800', tz='US/Pacific')
>>> tzinfo = ts.tzinfo
>>> ts = ts.replace(tzinfo=None)
>>> ts + pd.DateOffset(weekday=0)
Timestamp('2013-03-11 00:00:00', tz=None)
>>> ts = ts + pd.DateOffset(weekday=0)
>>> newts = tzinfo.localize(ts)
>>> pd.Timestamp(newts)
Timestamp('2013-03-11 00:00:00-0700', tz='US/Pacific')

[jreback]

@rockg is this true ONLY for a dst transition? (eg. should it always be done)?

[rockg]

@jreback Ideally only for offsets that span dst transitions, but I don't know how that will be known ahead of time (or at least in an efficient manner as you would basically need to do the operation twice). Localizing and unlocalizing are optimized for DatetimeIndex so I think doing it always makes sense in that case, but there will be some new plumbing. This will only fix his last two test cases.

[jreback]

ok....will put you down for this! if you want to try for 0.13 ok....but would need by next week

[rockg]

Okay. I probably won't get to it seriously until early next week so probably not for the release, but shortly after.

[jreback]

gr8

[adamgreenhall]

Actually I'm not sure about this solution method. It fails in the case where we are just adding hours over a DST transition:

def test_hours_add_fall_back():
    # fall back adds an extra hour. so 10pm + 8hrs should be 5am
    t = pd.Timestamp("2013-11-03 22:00", tz=tz) + pd.DateOffset(hours=8)
    assert(t.hour == 5)


def test_hours_add_spring_forward():
    # spring forward looses an hour. so 10pm + 8hrs should be 7am
    t = pd.Timestamp("2013-03-01 22:00", tz=tz) + pd.DateOffset(hours=8)
    assert(t.hour == 7)

@rockg thoughts?