Series[categorical] median raises, but DataFrame doesn't

[TomAugspurger]

This is a bit strange.

In [3]: pd.DataFrame({"A": pd.Categorical([1, 2, 2, 2, 3])})
Out[3]:
   A
0  1
1  2
2  2
3  2
4  3

In [4]: df = pd.DataFrame({"A": pd.Categorical([1, 2, 2, 2, 3])})

In [5]: df.median()
Out[5]:
A    2.0
dtype: float64

In [6]: df.A.median()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-6-15196eaacc89> in <module>()
----> 1 df.A.median()

~/sandbox/pandas/pandas/core/generic.py in stat_func(self, axis, skipna, level, numeric_only, **kwargs)
   9587                                       skipna=skipna)
   9588         return self._reduce(f, name, axis=axis, skipna=skipna,
-> 9589                             numeric_only=numeric_only)
   9590
   9591     return set_function_name(stat_func, name, cls)

~/sandbox/pandas/pandas/core/series.py in _reduce(self, op, name, axis, skipna, numeric_only, filter_type, **kwds)
   3220         return delegate._reduce(op=op, name=name, axis=axis, skipna=skipna,
   3221                                 numeric_only=numeric_only,
-> 3222                                 filter_type=filter_type, **kwds)
   3223
   3224     def _reindex_indexer(self, new_index, indexer, copy):

~/sandbox/pandas/pandas/core/arrays/categorical.py in _reduce(self, op, name, axis, skipna, numeric_only, filter_type, **kwds)
   2065         if func is None:
   2066             msg = 'Categorical cannot perform the operation {op}'
-> 2067             raise TypeError(msg.format(op=name))
   2068         return func(numeric_only=numeric_only, **kwds)
   2069

TypeError: Categorical cannot perform the operation median

Anyone know whether that's intentional?

[jreback]

median is like min/max it can only work on an ordered categorical

Dataframe is prob coercing to object?

[JarnoRFB]

Apparently there is some coercion going on. If I do the same with object categories df.median() just returns an empty series.

cat_ordered_series = pd.Series(
    pd.Categorical(['a', 'b', 'c', 'a'], categories=['b', 'c', 'a'],
                         ordered=True)
)
# cat_ordered_series.median() # raises TypeError
df = pd.DataFrame({'a': cat_ordered_series})
df.median()

Out: Series([], dtype: float64)

This really seems to be a little inconsistent. I think median calculations on unordered categories should always be omitted (when in DataFrames) or raise TypeError (when in Series). For ordered categories I am actually not sure, as the median can be computed, but is often not very meaningful. However, I think it is a bit confusing if categories are coerced to numbers.

[jbrockmendel]

This is, surprisingly, the "correct" behavior with numeric_only=None. In DataFrame._reduce this goes through the frame_apply path and operates column-wise. When it operates on that column it ignores the TypeError that is raised, and you end up getting an empty result back.

You'll get the expected also-raising if you do df.median(numeric_only=False)

xref #28900 on getting rid of this footgun.