Handling exceptions within a `@contextmanager` function doesn't clear `sys.exception()`

[pR0Ps]

Bug report

Bug description:

The issue

I would expect that handling exceptions within a contextlib.contextmanager-created function would work in the same way as other functions and clear the sys.exception() after an error is handled.

import contextlib
import sys

def p(msg):
    print(msg, repr(sys.exception()), sep=": ")

def ctx_gen():
    p("before yield")
    try:
        yield
    except:
        p("during handling")
    p("after handling")

ctx = contextlib.contextmanager(ctx_gen)

with ctx():
    1/0

The above example prints:

before yield: None
during handling: ZeroDivisionError('division by zero')
after handling: ZeroDivisionError('division by zero')

Whereas since the error was handled by the except: block, my expectation was:

before yield: None
during handling: ZeroDivisionError('division by zero')
after handling: None

Just working as designed?

From doing some digging, it seems like this is happening because the exception is still being handled by the _GeneratorContextManager.__exit__ function (added by the @contextlib.contextmanager decorator) that's driving the ctx_gen generator, even after the ctg_gen has handled it.

The following is a very rough approximation of how @contextlib.contextmanager drives ctx_gen:

c = ctx_gen()
next(c)  # __enter__()
try:
    # code inside the with block
    1/0
except Exception as e:  # __exit__(typ, exc, tb) for e
    # throw exception into generator and expect to run to end
    try:
        c.throw(e) 
    except StopIteration:
        pass
else:  # __exit__(None, None, None)
    # expect to run to end
    try:
        next(e)
    except StopIteration:
        pass

Running the above (including the definitions from the first code block) also prints:

before yield: None
during handling: ZeroDivisionError('division by zero')
after handling: ZeroDivisionError('division by zero')

In the above code, it's more clear that the exception is still being handled by the except Exception as e: block until c.throw() returns/raises, which only happens after the generator exits. Therefore the exception is still being handled the entire time ctx_gen is running all the code after the first yield.

The fix?

Even though this behavior looks to be technically correct, it still seems unexpected and a bit of an abstraction leak.

Is this something that can be/should be fixed? Or should the behavior just be documented?

CPython versions tested on:

3.8, 3.9, 3.10, 3.11, CPython main branch

Operating systems tested on:

macOS

Linked PRs

• gh-111375: Fix handling of exceptions within @contextmanager-decorated functions #111676
• gh-111375: Use NULL in the exception stack to indicate an exception was handled #113302

[ericsnowcurrently]

CC @ncoghlan

[iritkatriel]

This is a nice bug - everything is working as designed but the result is indeed unexpected.

I don't think it can be fixed in contextlib without changes to the interpreter, and if I'm right it's unlikely that this will use case will justify such changes.

Either way, for now at least we should mention this as a quirk of contextlib. @pR0Ps would you like to propose a doc patch?

[pR0Ps]

I was messing around with this and after I wired up a C API call to PyErr_SetHandledException(Py_None) just before the self.gen.throw line in the following code, I get the "expected" result.

Before:

cpython/Lib/contextlib.py

Lines 161 to 163 in 230e8e9

	try:
	self.gen.throw(value)
	except StopIteration as exc:

After:

try:
    __clear_current_exc()  # defined in C as: { PyErr_SetHandledException(Py_None); return Py_None; }
    self.gen.throw(value)
except StopIteration as exc:

This essentially amounts to running Python 2's sys.exc_clear() just before throwing the exception into the generator. While the generator still has to handle the exception that was thrown into it, it's doing it with no current exception context. This means that once it handles that thrown exception, there is no longer any exception being handled, making sys.exception() return None which matches what a user would expect.

With this change, the result of running the test case from above matches my expected result:

before yield: None
during handling: ZeroDivisionError('division by zero')
after handling: None

At the moment, my implementation is just a proof of concept that I quickly hacked together, but I'd be willing to clean it up, fix the edge cases, write tests, etc. if this is something that would be considered acceptable to add in some fashion and not too hacky.

[iritkatriel]

Ah yes, that might work. You'd need to restore the exception after throw() returns. Make sure you have a test for when this whole thing is nested in another except block (and check the value of sys.exception() after the nested one).