PERF: regression in DataFrame reduction ops performance

[jorisvandenbossche]

From https://pandas.pydata.org/speed/pandas/#stat_ops.FrameOps.time_op?Cython=0.29.21&Cython=0.29.16&p-op='sum'&p-dtype='int'&p-axis=0&commits=3a043f2d-4c03d07b&x-axis-scale=date

Reproducer:

values = np.random.randn(100000, 4)  
df = pd.DataFrame(values).astype("int")
%timeit df.sum()

increased with a factor 2 to 3 x somewhere the last days.

[ukarroum]

Would like to try working on that if possible.

[ukarroum]

take

[jbrockmendel]

#35881 is a candidate, though thats a much bigger impact than id expect

[jorisvandenbossche]

Checking out that commit vs the one before, seems to confirm that #35881 is indeed the cause of the slowdown

[jorisvandenbossche]

I did a quick profile, and commented on #35881 about some observations

[jorisvandenbossche]

@ukarroum especially #35881 (comment) is something you could test

[ukarroum]

Can confirm that by using self._iter_column_arrays() instead of self.dtypes we have significant performance improvement :

With self.dtypes :

In [8]: values = np.random.randn(100000, 4)   
   ...: df = pd.DataFrame(values).astype("int") 
   ...: %timeit df.sum() 
714 µs ± 21 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

With self._iter_column_arrays()

In [4]: values = np.random.randn(100000, 4)   
   ...: df = pd.DataFrame(values).astype("int") 
   ...: %timeit df.sum() 
477 µs ± 8.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Gonna submit a PR soon.