CLEAN: Reconcile types in pandas._typing.py and pandas-stubs._typing.pyi

[Dr-Irv]

There are some inconsistencies in types in pandas._typing.py and pandas-stubs._typing.pyi . These should be reconciled.

In addition, types that are only in the stubs should probably be private with a leading underscore _

[bashtage]

Are there any tools that will help keep stubs in sync with pandas? When I went through io/stata.py there were a lot of changes since these were originally collatted.

[Dr-Irv]

Are there any tools that will help keep stubs in sync with pandas? When I went through io/stata.py there were a lot of changes since these were originally collatted.

Not that I'm aware of. Maybe stubtest could help

[twoertwein]

Might not make sense to fix all of them (some exist on main but not in 1.4.x) but it gives a quick overview what could be changed poe stubtest | grep "pandas\._typing"

pandas._typing.AggFuncTypeDict is not a type
pandas._typing.Axes is not a Union
pandas._typing.AxisType is not present at runtime
pandas._typing.BaseBuffer.init is inconsistent, stub does not have *args argument "args"
pandas._typing.Buffer is not present at runtime
pandas._typing.CompressionDict is not a type
pandas._typing.ConvertersArg is not present at runtime
pandas._typing.DateTimeErrorChoices is not present at runtime
pandas._typing.DatetimeLikeScalar is not a TypeVar
pandas._typing.DtypeNp is not present at runtime
pandas._typing.FileOrBuffer is not present at runtime
pandas._typing.FilePathOrBuffer is not present at runtime
pandas._typing.FilePathOrBytesBuffer is not present at runtime
pandas._typing.FrameOrSeries is not present at runtime
pandas._typing.FrameOrSeriesUnion is not present at runtime
pandas._typing.FuncType is not a recognised type alias
pandas._typing.Function is not present at runtime
pandas._typing.GroupByObject is not present at runtime
pandas._typing.GroupByObjectNonScalar is not present at runtime
pandas._typing.HashableT is not present at runtime
pandas._typing.IgnoreRaise is not present at runtime
pandas._typing.IndexType is not present at runtime
pandas._typing.IndexingInt is not present at runtime
pandas._typing.IntervalClosedType is not present at runtime
pandas._typing.KeysArgType is not present at runtime
pandas._typing.Label is not present at runtime
pandas._typing.ListLike is not present at runtime
pandas._typing.MaskType is not present at runtime
pandas._typing.ParseDatesArg is not present at runtime
pandas._typing.ReadBuffer.init is inconsistent, stub does not have *args argument "args"
pandas._typing.S1 is not present at runtime
pandas._typing.ScalarT is not present at runtime
pandas._typing.SeriesAxisType is not present at runtime
pandas._typing.StrLike is not present at runtime
pandas._typing.T1 is not present at runtime
pandas._typing.T2 is not present at runtime
pandas._typing.WriteBuffer.init is inconsistent, stub does not have *args argument "args"
pandas._typing.XMLParsers is not a Union
pandas._typing.all is not present at runtime
pandas._typing.np_ndarray_anyint is not present at runtime
pandas._typing.np_ndarray_bool is not present at runtime
pandas._typing.np_ndarray_int is not present at runtime
pandas._typing.np_ndarray_int16 is not present at runtime
pandas._typing.np_ndarray_int32 is not present at runtime
pandas._typing.np_ndarray_int64 is not present at runtime
pandas._typing.np_ndarray_int8 is not present at runtime
pandas._typing.np_ndarray_str is not present at runtime
pandas._typing.np_ndarray_uint is not present at runtime
pandas._typing.np_ndarray_uint16 is not present at runtime
pandas._typing.np_ndarray_uint32 is not present at runtime
pandas._typing.np_ndarray_uint64 is not present at runtime
pandas._typing.np_ndarray_uint8 is not present at runtime
pandas._typing.num is not present at runtime
pandas._typing.type_t is not a type

[Dr-Irv]

Can this tell us which ones are in pandas._typing.py, but not pandas._typing.pyi ?

Also would be best if we could use what is in main from pandas just to keep things moving forward.

[twoertwein]

Can this tell us which ones are in pandas._typing.py, but not pandas._typing.pyi ?

I think so, would probably need to remove --ignore-missing-stub from the stubtest call.

Also would be best if we could use what is in main from pandas just to keep things moving forward.

stubtest will use whichever version of pandas that is installed. Currently, that is 1.4.x. I think pandas has nightly builds, we could install that in pyproject.toml instead of 1.4.x: but then we might get some "troubles" with pytests, more deprecation warnings (before 1.5) and breaking changes (after 1.5 is released).

[twoertwein]

Can this tell us which ones are in pandas._typing.py, but not pandas._typing.pyi ?

I wrote some quick&ugly ast parsing code to check when pandas-stubs has annotations but pandas has none (I have been using it for pandas-dev/pandas#47827 and similar PRs). Could extend that, but stubtest without --ignore-missing-stub would be a better choice.

[Dr-Irv]

stubtest will use whichever version of pandas that is installed. Currently, that is 1.4.x. I think pandas has nightly builds, we could install that in pyproject.toml instead of 1.4.x: but then we might get some "troubles" with pytests, more deprecation warnings (before 1.5) and breaking changes (after 1.5 is released).

Maybe you could just figure out a way to run stubtest (but not mypy, pytest, etc.) against the current version in pandas/main so we can start reconciling the types. There is good typing work in pandas that we can pull over to the stubs without making the stubs incorrect for 1.4.x

[twoertwein]

Could have a step ala "remove pandas 1.4x. and install pandas nightly" with an appropriate rollback.