`StopIteration` in a generator in a thread hanging on asyncio

[NewUserHa]

Bug report

Bug description:

import asyncio

def a():
    raise StopIteration

try:
    await asyncio.to_thread(a)
except:
    print(3.12.0)

other cases: agronholm/anyio#477

CPython versions tested on:

3.11, 3.12

Operating systems tested on:

Windows

Linked PRs

• gh-112182: Replace StopIteration with RuntimeError for future #113220
• [3.12] gh-112182: Replace StopIteration with RuntimeError for future (GH-113220) #123033

[terryjreedy]

Please post working code.
print(3.12.0) is a SyntaxError.
With that fixed, 'await' outside an async function is also a syntax error.
With that fixed I still get an error.

import asyncio

def a():
    raise StopIteration

async def f():
    try:
        await ayncio.to_thread(a)
    except:
        print('done')

f()

results in RuntimeWarning: coroutine 'f' was never awaited.
(Afaik, without yield, a is not a generator function.)

[NewUserHa]

That is the working code in notebook, except the 3.12.0 was trying to match the version number of the used python.
Using StopIteration instead of yield is for simplifying the code, and also report that this exception can also cause the issue.

Anyway, here are the working codes without notebook:

• you will see it never exits.

import asyncio


def a():
    raise StopIteration


async def main():
    try:
        await asyncio.to_thread(a)
    except:
        print(3.12)

asyncio.run(main())

• the yeild one

import asyncio


def a():
    for _ in [1, 2]:
        yield _


async def main():
    gen = a()
    while True:
        try:
            await asyncio.to_thread(next, gen)
        except:
            print(3.12)
            break

asyncio.run(main())

Also, in the related link, it has more example cases

[terryjreedy]

I just noticed that my code had a spelling error masked by the overly broad except.
With the spelling and the call to f (or main), the following (in IDLE with Win 10)

Exception in callback _chain_future.<locals>._set_state(<Future pendi...ask_wakeup()]>, <Future at 0x...StopIteration>) at C:\Programs\Python313\Lib\asyncio\futures.py:381
handle: <Handle _chain_future.<locals>._set_state(<Future pendi...ask_wakeup()]>, <Future at 0x...StopIteration>) at C:\Programs\Python313\Lib\asyncio\futures.py:381>
Traceback (most recent call last):
  File "C:\Programs\Python313\Lib\asyncio\events.py", line 84, in _run
    self._context.run(self._callback, *self._args)
  File "C:\Programs\Python313\Lib\asyncio\futures.py", line 383, in _set_state
    _copy_future_state(other, future)
  File "C:\Programs\Python313\Lib\asyncio\futures.py", line 359, in _copy_future_state
    dest.set_exception(_convert_future_exc(exception))
TypeError: StopIteration interacts badly with generators and cannot be raised into a Future

and does indeed 'hang' (no >>> prompt appears). Hit ^C and

Traceback (most recent call last):
  File "C:\Programs\Python313\Lib\asyncio\runners.py", line 118, in run
    return self._loop.run_until_complete(task)
  File "C:\Programs\Python313\Lib\asyncio\base_events.py", line 664, in run_until_complete
    return future.result()
  File "F:\dev\tem\tem2.py", line 8, in main
    await asyncio.to_thread(a)
  File "C:\Programs\Python313\Lib\asyncio\threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "F:\dev\tem\tem2.py", line 12, in <module>
    asyncio.run(main())
  File "C:\Programs\Python313\Lib\asyncio\runners.py", line 194, in run
    return runner.run(main)
  File "C:\Programs\Python313\Lib\asyncio\runners.py", line 123, in run
    raise KeyboardInterrupt()
KeyboardInterrupt

and a prompt appears. I don't know enough about asyncio to say more, but I add the label to ping people who do.

[gvanrossum]

My first response is "don't do that". That's why it prints such an elaborate warning.

Looking into the machinery involved, maybe there could be a try/except around the else block in _copy_future_state() (in asyncio/futures.py) that if the whole thing fails just calls dest.cancel(). That way the caller at least won't hang.

[gvanrossum]

The failure should also be logged. Perhaps you can find the loop where to log it via dest._loop. Make sure that a failure to log doesn't crash.

[NewUserHa]

So the PR should look like:

from .log import logger
def _copy_future_state(source, dest):
    """Internal helper to copy state from another Future.

    The other Future may be a concurrent.futures.Future.
    """
    assert source.done()
    if dest.cancelled():
        return
    assert not dest.done()
    if source.cancelled():
        dest.cancel()
    else:
        try:
            exception = source.exception()
            if exception is not None:
                dest.set_exception(_convert_future_exc(exception))
            else:
                result = source.result()
                dest.set_result(result)
        except:
            dest.cancel()
            # dest._loop.
            logger.error('Exception in default exception handler',
                        exc_info=True)

?

it seems that it's dest.set_exception(_convert_future_exc(exception)) raised error, and which is at

    def set_exception(self, exception):
        """Mark the future done and set an exception.

        If the future is already done when this method is called, raises
        InvalidStateError.
        """
        if self._state != _PENDING:
            raise exceptions.InvalidStateError(f'{self._state}: {self!r}')
        if isinstance(exception, type):
            exception = exception()
        if type(exception) is StopIteration:
            raise TypeError("StopIteration interacts badly with generators "
                            "and cannot be raised into a Future")
        self._exception = exception
        self._exception_tb = exception.__traceback__
        self._state = _FINISHED
        self.__schedule_callbacks()
        self.__log_traceback = True

why would set_exception() raise the error because of 'stopiteration' in python?

and what about changing the exception to runtimeerror like how the anyio does in its issue at the above first comment

[gvanrossum]

This surprised me too. Maybe the git history of those lines has a clue.

[NewUserHa]

It seems that this line exists after 3.5.2rc1 (not in 3..5.1),
and its commit is 1bd0307 and it referred to an invalid issue

[gvanrossum]

That is a bpo issue: https://bugs.python.org/issue26221