BUG: in read_excel for mangled columns only the original/first column dtype is correct, col.N is not parsed correctly

[jeroenbrouwer]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

This requires setting up an excel file. Reproducible by setting 2 columns with equal headers like a and a row with values 1. Schematic:

a a
1 1

Then using:

df = pd.read_excel(file, dtype={'a': str})
assert type(df['a'][0]) == type(df['a.1'][0]) 

Problem description

The problem is that column a is parsed as a str and column a.1 is parsed as a float64.

Expected Output

Both a and a.1 should be parsed as str, such that the assertion is passed.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.7.4.final.0
python-bits : 64
OS : Darwin
OS-release : 19.4.0
machine : x86_64
processor : i386
byteorder : little
LC_ALL : en_US.UTF-8
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.0.5
numpy : 1.19.0
pytz : 2020.1
dateutil : 2.8.1
pip : 20.1.1
setuptools : 40.8.0
Cython : 0.29.20
pytest : 5.3.5
hypothesis : None
sphinx : 3.0.4
blosc : None
feather : None
xlsxwriter : 1.2.9
lxml.etree : 4.5.1
html5lib : None
pymysql : None
psycopg2 : 2.8.5 (dt dec pq3 ext lo64)
jinja2 : 2.11.1
IPython : 7.15.0
pandas_datareader: None
bs4 : 4.9.1
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : 4.5.1
matplotlib : 3.2.1
numexpr : None
odfpy : None
openpyxl : 3.0.3
pandas_gbq : None
pyarrow : None
pytables : None
pytest : 5.3.5
pyxlsb : None
s3fs : None
scipy : 1.4.1
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : 1.2.0
xlwt : None
xlsxwriter : 1.2.9
numba : 0.50.0

[simonjayhawkins]

Thanks @jeroenbrouwer for the report. without looking at the code, the coercion is probably being applied after the name mangling otherwise I would expected the following to raise

>>> pd.__version__
'1.1.0.dev0+2067.g2c3edaaaa'
>>>
>>> import tempfile
>>>
>>> df = pd.DataFrame([[1, 1]], columns=list("aa"))
>>> df
   a  a
0  1  1
>>>
>>> with tempfile.TemporaryFile() as fp:
...     df.to_excel(fp)
...     res = pd.read_excel(fp, dtype={"a": str})
...
>>> print(res)
   Unnamed: 0  a  a.1
0           0  1    1
>>>
>>> res.dtypes
Unnamed: 0     int64
a             object
a.1            int64
dtype: object
>>>
>>> with tempfile.TemporaryFile() as fp:
...     df.to_excel(fp)
...     res = pd.read_excel(fp, dtype={"a": str, "a.1": str})
...
>>> print(res)
   Unnamed: 0  a a.1
0           0  1   1
>>>
>>> res.dtypes
Unnamed: 0     int64
a             object
a.1           object
dtype: object
>>>

further investigation and PRs welcome.

[sharvanir]

@simonjayhawkins can I check it out to see if I can fix?

[sharvanir]

take

[jeroenbrouwer]

Any luck so far @sharvanir ?

[sharvanir]

hi @jeroenbrouwer looks like an issue with python's AST module, trying to find a workaround

[asishm]

read_excel uses the PythonParser behind the scene to convert the parsed data to a dataframe.

Simpler repro would be

In [1]: import pandas as pd

In [2]: data = '''a,a\n1,1'''

In [3]: from io import StringIO

In [4]: df = pd.read_csv(StringIO(data), dtype={'a': str}, mangle_dupe_cols=True, engine='python')

In [9]: df
Out[9]: 
   a  a.1
0  1    1

In [10]: df.dtypes
Out[10]: 
a      object
a.1     int64
dtype: object

engine='c' has the same behavior as the python engine currently.

If this behavior is changed at the PythonParser level, then I believe a corresponding change should happen in the CParserWrapper to remain consistent.

[sharvanir]

sorry, got confused, it's not an ast issue. I will unassign this as I am unable to put in time due to professional commitments.
Will get back if possible