DataFrame.groupby.fillna for axis=1, no filling happens

[prebenbang]

groupby.fillna not working along column axis.

The bug exists for 15.1 and 15.2
Works for 13.1

import pandas as pd
import numpy as np

print pd.__version__

test = pd.DataFrame(index=pd.MultiIndex.from_product([['value1','value2'],pd.date_range('2014-01-01','2014-01-06')]),
                    columns=pd.Index(['1','2'], name='id'))
test['1'] = [np.nan, 1, np.nan, np.nan, 11, np.nan, np.nan, 2, np.nan, np.nan, 22, np.nan]
test['2'] = [np.nan, 3, np.nan, np.nan, 33, np.nan, np.nan, 4, np.nan, np.nan, 44, np.nan]
testT = test.T

print '*' * 30
print test
print '*' * 30
print test.groupby(level=0, axis=0).fillna(method='ffill')
print '*' * 30
print testT
print '*' * 30
print testT.groupby(level=0, axis=1).fillna(method='ffill')
print '*' * 30

[jreback]

a buggie. The generated arguments to the wrapped function (fillna in this case), were not detected properly that the real axis was 1 (and not the default of 0), as the arguments were set too early. Fixed in #9222 .

[behzadnouri]

@jreback the added test already passes on master:

In [1]: pd.__version__
Out[1]: '0.15.2-68-ge2b014c'

In [2]: df = DataFrame(index=pd.MultiIndex.from_product([['value1','value2'],
   ...:                date_range('2014-01-01','2014-01-06')]),
   ...:                columns=Index(['1','2'], name='id'))

In [3]: df['1'] = [np.nan, 1, np.nan, np.nan, 11, np.nan, np.nan, 2, np.nan, np.nan, 22, np.nan]

In [4]: df['2'] = [np.nan, 3, np.nan, np.nan, 33, np.nan, np.nan, 4, np.nan, np.nan, 44, np.nan]

In [5]: expected = df.groupby(level=0, axis=0).fillna(method='ffill')

In [6]: result = df.T.groupby(level=0, axis=1).fillna(method='ffill').T

In [7]: assert_frame_equal(result, expected)

In [8]: 

[jreback]

doesn't look like it

In [6]: expected
Out[6]: 
id                  1   2
value1 2014-01-01 NaN NaN
       2014-01-02   1   3
       2014-01-03   1   3
       2014-01-04   1   3
       2014-01-05  11  33
       2014-01-06  11  33
value2 2014-01-01 NaN NaN
       2014-01-02   2   4
       2014-01-03   2   4
       2014-01-04   2   4
       2014-01-05  22  44
       2014-01-06  22  44

In [7]: result
Out[7]: 
id                  1   2
value1 2014-01-01 NaN NaN
       2014-01-02   1   3
       2014-01-03 NaN NaN
       2014-01-04 NaN NaN
       2014-01-05  11  33
       2014-01-06 NaN NaN
value2 2014-01-01 NaN NaN
       2014-01-02   2   4
       2014-01-03 NaN NaN
       2014-01-04 NaN NaN
       2014-01-05  22  44
       2014-01-06 NaN NaN 

In [8]: pd.__version__
Out[8]: '0.15.2-64-g128ce85'

[behzadnouri]

this can be a platform issue; this is what i get:

In [10]: df
Out[10]: 
id                  1   2
value1 2014-01-01 NaN NaN
       2014-01-02   1   3
       2014-01-03 NaN NaN
       2014-01-04 NaN NaN
       2014-01-05  11  33
       2014-01-06 NaN NaN
value2 2014-01-01 NaN NaN
       2014-01-02   2   4
       2014-01-03 NaN NaN
       2014-01-04 NaN NaN
       2014-01-05  22  44
       2014-01-06 NaN NaN

In [11]: df.T.groupby(level=0, axis=1).fillna(method='ffill').T
Out[11]: 
id                  1   2
value1 2014-01-01 NaN NaN
       2014-01-02   1   3
       2014-01-03   1   3
       2014-01-04   1   3
       2014-01-05  11  33
       2014-01-06  11  33
value2 2014-01-01 NaN NaN
       2014-01-02   2   4
       2014-01-03   2   4
       2014-01-04   2   4
       2014-01-05  22  44
       2014-01-06  22  44

[jreback]

@behzadnouri hmm

what is your platform?

I am running macosx. This had to do with detection in the make_wrapper of the passed args. (and axis=0 was being passed. Not sure how that could work otherwise. IOW it fills but on the wrong axis.

[behzadnouri]

In [10]: pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.4.2.final.0
python-bits: 64
OS: Linux
OS-release: 3.17.6-1-ARCH
machine: x86_64
processor: 
byteorder: little
LC_ALL: None
LANG: en_US.utf8

pandas: 0.15.2-68-ge2b014c
nose: 1.3.4
Cython: 0.21.2
numpy: 1.9.1
scipy: 0.14.1
statsmodels: 0.6.0.dev-b52bc09
IPython: 2.3.1
sphinx: None
patsy: 0.2.1+dev
dateutil: 2.3
pytz: 2014.10
bottleneck: 0.9.0dev
tables: None
numexpr: 2.4
matplotlib: 1.4.2
openpyxl: 2.1.4
xlrd: None
xlwt: None
xlsxwriter: None
lxml: 3.4.1
bs4: 4.3.2
html5lib: None
httplib2: 0.9
apiclient: None
rpy2: None
sqlalchemy: None
pymysql: None
psycopg2: None

[jreback]

I just tried this on 3.4 and it works! (I am on 2.7).

[jreback]

@prebenbang you are on 2.7? can you do a pd.show_versions()

[prebenbang]

Yup, I'm on 2.7.