Allow @overload outside stubs

[bdarnell]

Mypy does not consider the stub file for the module it is currently type-checking (and doing so was apparently considered and rejected recently). However, @overload as defined in the PEP is only allowed to appear in stubs, which implies a requirement for type checkers to consider a module's own stubs when type checking it. Otherwise, the ability to type check a module which defines an overloaded function is limited.

My motivating example:

@overload
def utf8(s: None) -> None: ...
@overload
def utf8(s: AnyStr) -> bytes: ...

This could be expressed as def utf8(s: Optional[AnyStr]) -> Optional[bytes], but I'm trying to capture the fact that if the argument is not None then the result is also guaranteed not to be None, so it doesn't introduce unnecessary Optional-ness. (Is there a way to express this with a TypeVar? I don't see one). I can put the overloads in a stub for the benefit of external callers of this module, but then I need to put all the type information for this module in the stub and can't type check it internally (I could duplicate all the non-overload type info in the module itself and get partial coverage). I could also put overloaded functions in their own tiny modules to minimize the impact, but that just feels silly.

[gvanrossum]

I think it can be done with typevars: AnyStr is defined as TypeVar('AnyStr', str, bytes). It's not special in any way though. You should be able to define your own typevar as OptAnyStr = TypeVar('OptAnyStr', None, str, bytes). I think we're not in principle against eventually considering the .pyi and .py files together, but it would be a lot of work to add this to mypy, and your use case (@overload) is pretty much the only one. (I vaguely recall another issue discussing this, but I can't find it here. Maybe it's in the typeshed repo or even in typehinting, which is for PEP 484 and the PyPI version of typing.py.)

[JukkaL]

I'm sure there's a better way to support @overload than having a separate .pyi file (such as the approach discussed in python/typing#72). Separate .pyi files will be painful to maintain, and they don't provide most of the readability gains of inline annotations.

[bdarnell]

I don't see how OptAnyStr would help. Note that there are two different unions involved: (Union[bytes, unicode, None]) -> Union[bytes, None]. I want to constrain the return type as a function of the input type, but it can't just pick the same type on both sides because of the (unicode) -> bytes case.

I'd be happy with a non-stub way to express overloads (and would prefer it to a requirement for separate stub files to use this feature). The # type: comment syntax could support this relatively easily and wouldn't carry the implications of runtime dispatch like a decorator would.

Overloads that can't be satisfied with TypeVars may be rare, but when they come up they can be significant (that's why the feature is in the PEP). In this case, tornado.escape.utf8 is used all over Tornado, and I'd have to add dozens of explicit casts if I had to use the union declaration instead of the overloaded version.

The interaction between stub files and source files is generally awkward. I just realized that the type annotations I've been adding to Tornado source files aren't actually used outside those files because there are stubs for Tornado in typeshed (mypy --verbose should report where it's loading type information from). Assuming I release a version of Tornado with inline type annotations I'm going to need to be able to make sure that those types take precedence over the versions in typeshed.

[gvanrossum]

Wow, you're really bleeding edge here. Great to see you exploring this and helping us find the gray areas. Short term, the only solution I can think of for utf8() would be to move it to a separate file and have a stub for that file containing the @overload declarations. That's awkward, but it can work with today's mypy. Longer term, this example suggests that we really need to thing harder about allowing @overload in non-stub files. That's a PEP 484 change and we should discuss it on python-dev, but we might as well first prepare a proposal. I've started laying out the problem in python/typing#175 -- let's continue the discussion there. Regarding inline annotations vs typeshed, that's another unsolved problem. I've opened python/typeshed#52 for it, but I frankly don't know how to solve it. Please join those discussions!

[gvanrossum]

Actually, as I wrote in python/typeshed#52, the stubs vs. real code problem is easily solved by setting MYPYPATH to include the version of Tornado you're using -- the stubs in typeshed are searched last. The solution for @overload utf8() is to change the PEP, which I will do shortly.

[gvanrossum]

The PEP is now updated. This issue should stay open until it's been implemented in mypy.