Skip to content

BUG: Nullable integer dtype leads to a fragmentation PerformanceWarning #44098

Closed
@epizut

Description

@epizut

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the master branch of pandas.

Reproducible Example

import numpy as np
import pandas as pd

df = pd.DataFrame(np.random.randint(0,100,size=(100, 100)), dtype='Int32')
df.reset_index()

Issue Description

When using the new nullable Integer dtype, pandas seams to generate one IntegerArray per columns.
This will inevitably leads to a fragmentation performance warning while using more than 100 columns.

df = pd.DataFrame(np.random.randint(0,100,size=(100, 100)), dtype='Int32')
df.reset_index() # Warn
>>> PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead.  To get a de-fragmented frame, use `newframe = frame.copy()`
  df.reset_index()

Of course this is due to an high number of fragments:

df._data.nblocks
>>> 100

Expected Behavior

Maybe we could have a proper nullable integer matrix representation

Installed Versions

INSTALLED VERSIONS ------------------ commit : 945c9ed python : 3.9.6.final.0 python-bits : 64 OS : Windows OS-release : 10 Version : 10.0.18363 machine : AMD64 processor : Intel64 Family 6 Model 85 Stepping 4, GenuineIntel byteorder : little LC_ALL : None LANG : None LOCALE : English_United States.1252

pandas : 1.3.4
numpy : 1.21.1
pytz : 2021.1
dateutil : 2.8.2
pip : 21.2.4
setuptools : 56.0.0
Cython : None
pytest : 6.2.4
hypothesis : None
sphinx : 4.1.2
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.0.1
IPython : 7.26.0
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : 2021.07.0
fastparquet : None
gcsfs : None
matplotlib : 3.4.2
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 5.0.0
pyxlsb : None
s3fs : None
scipy : 1.7.1
sqlalchemy : 1.4.22
tables : None
tabulate : None
xarray : 0.19.0
xlrd : None
xlwt : None
numba : None

Metadata

Metadata

Assignees

No one assigned

    Labels

    ExtensionArrayExtending pandas with custom dtypes or arrays.

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions