BUG: pd.to_datetime(infer_datetime_format=True) drops timezone

[scottyhq]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

dates = ['2020-05-01T18:04:03Z', '2020-04-28T17:54:06Z', '2020-04-26T18:04:09Z']
print(pd.to_datetime(dates, infer_datetime_format=True)) #drops UTC
print(pd.to_datetime(dates, infer_datetime_format=False)) #works!

dates = ['2020-05-01T18:04:03.0Z', '2020-04-28T17:54:06.0Z', '2020-04-26T18:04:09.0Z']
print(pd.to_datetime(dates, infer_datetime_format=True)) #works!

Output:

DatetimeIndex(['2020-05-01 18:04:03', '2020-04-28 17:54:06',
               '2020-04-26 18:04:09'],
              dtype='datetime64[ns]', freq=None)
DatetimeIndex(['2020-05-01 18:04:03+00:00', '2020-04-28 17:54:06+00:00',
               '2020-04-26 18:04:09+00:00'],
              dtype='datetime64[ns, UTC]', freq=None)
DatetimeIndex(['2020-05-01 18:04:03+00:00', '2020-04-28 17:54:06+00:00',
               '2020-04-26 18:04:09+00:00'],
              dtype='datetime64[ns, UTC]', freq=None)

Problem description

Using infer_datetime_format=True drops the timezone info (UTC) if datestrings do not have decimal seconds as shown in the example above. '2020-05-01T18:04:03Z' is a valid ISO 8601 datetime (https://tools.ietf.org/html/rfc3339#section-5.6).

Expected Output

dates = ['2020-05-01T18:04:03Z', '2020-04-28T17:54:06Z', '2020-04-26T18:04:09Z']
print(pd.to_datetime(dates, infer_datetime_format=True))
#DatetimeIndex(['2020-05-01 18:04:03+00:00', '2020-04-28 17:54:06+00:00',
#               '2020-04-26 18:04:09+00:00'],
#             dtype='datetime64[ns, UTC]', freq=None)

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 2cb9652
python : 3.8.8.final.0
python-bits : 64
OS : Linux
OS-release : 4.14.177-139.253.amzn2.x86_64
Version : #1 SMP Wed Apr 29 09:56:20 UTC 2020
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : C.UTF-8
LANG : C.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.2.4
numpy : 1.20.2
pytz : 2021.1
dateutil : 2.7.5
pip : 20.3.4
setuptools : 49.6.0.post20210108
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : 1.10.2
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : 2.8.6 (dt dec pq3 ext lo64)
jinja2 : 2.11.3
IPython : 7.22.0
pandas_datareader: None
bs4 : 4.9.3
bottleneck : 1.3.2
fsspec : 2021.04.0
fastparquet : None
gcsfs : 0.7.2
matplotlib : 3.4.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 3.0.0
pyxlsb : None
s3fs : 0.6.0
scipy : 1.6.2
sqlalchemy : 1.4.9
tables : None
tabulate : 0.8.9
xarray : 0.17.0
xlrd : None
xlwt : None
numba : 0.53.1

[ishansmishra]

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.to_datetime.html
Towards the end of the page, it mentions "Passing infer_datetime_format=True can often-times speedup a parsing if its not an ISO8601 format exactly, but in a regular format."
I don't know if this information helps or not, I'm new to open source in general :)

[i-aki-y]

This problem seems to happen the following codes in datetimes.py:

    if infer_datetime_format and format is None:
        format = _guess_datetime_format_for_array(arg, dayfirst=dayfirst)

    if format is not None:
        # There is a special fast-path for iso8601 formatted
        # datetime strings, so in those cases don't use the inferred
        # format because this path makes process slower in this
        # special case
        format_is_iso8601 = format_is_iso(format)
        if format_is_iso8601:
            require_iso8601 = not infer_datetime_format
            format = None

In the above code, the format_is_iso seems not to recognize the format that ends with 'Z', while the guess_datetime_format function used to infer the format can recognize the 'Z' and return the format include 'Z'.
So the inferred format string that has 'Z' can be wrongly navigated to the non-utc parse function.

I think fixing the format_is_iso function seems to solve the problem.

Ex.

def format_is_iso(f: str) -> bint:
    """
    Does format match the iso8601 set that can be handled by the C parser?
    Generally of form YYYY-MM-DDTHH:MM:SS - date separator can be different
    but must be consistent.  Leading 0s in dates and times are optional.
    """
    iso_template = '%Y{date_sep}%m{date_sep}%d{time_sep}%H:%M:%S.%f'.format
    excluded_formats = ['%Y%m%d', '%Y%m', '%Y']

+    if (f is not None) and (f[-2:] in ["SZ", "fZ"]):
+        # remove last 'Z'
+        f = f[:-1]

    for date_sep in [' ', '/', '\\', '-', '.', '']:
        for time_sep in [' ', 'T']:
            if (iso_template(date_sep=date_sep,
                             time_sep=time_sep
                             ).startswith(f) and f not in excluded_formats):
                return True
    return False

If this is true, let me know. I will make PR.