PEP 667 + PEP 709 segfaults from accessing closure variables bound by any inlined comprehensions

[bswck]

Bug report

Bug description:

The segfault described here is related to PEP-667 and PEP-709.

I think I experienced a couple of similar but not same segfaults, and this here is just one of a few of them. I might be wrong though.
I'd like to start working on fixing this in order to possibly find more unhappy paths.

Reproduction

Here are a bunch of reproductions:

def f():
    lambda: k
    k: int
    [k for k in [0] if locals()]

f()

def f():
    lambda: k
    k = [sys._getframe(0).f_locals["k"] for k in [0]]

f()

def f():
    lambda: k
    k = 1
    [eval("") for k in [0]]

f()

@alexmojaki also found

def f():
    lambda: k
    k = 1
    [locals() for k in [0]]

f()

def f():
    lambda: k
    k: int
    [locals() for k in [0]]

f()

This one shows the problem the best:

def fetch_locals_proxy():
    # anything that fetches PEP 667 FrameLocalsProxy contents
    # causes the crash
    sys._getframe(1).f_locals.items()

def f():
    lambda: k
    k = 1
    [fetch_locals_proxy() for k in [0]]

f()

Traceback

Fatal Python error: Segmentation fault

Current thread 0x00007fd059596740 (most recent call first):
  File "/tmp/repro/dump-1.py", line 9 in fetch_locals_proxy
  File "/tmp/repro/dump-1.py", line 14 in f
  File "/tmp/repro/dump-1.py", line 17 in <module>
Segmentation fault (core dumped)

Conclusions

1. There must be a statement that allows to create a closure for a variable, for instance k, in a scope ("target scope"). It's either k = ... or k: int in the repros.

2. There must be an inlined comprehension that binds k as target (e.g. k = 1; [x for x in [0] if locals()] doesn't cause segfault).

3. After binding the variable during the evaluation of the inlined comprehension, something must fetch the contents of the PEP 667 proxy of the target scope.

4. The target scope must be any optimized scope (not confirmed for generator expressions), since only those involve PEP 667 FrameLocalsProxy. That's the reason why the crash doesn't happen at a module/class level:

   class Spam:
       lambda: k
       k = [k for k in [0] if locals()]

   or on Python <3.13, before PEP 667 had been introduced.

5. The iterable traversed in the comprehension isn't special (the crash doesn't happen for certain objects, e.g. [None]¹, (...,), but happens for others, e.g. [0], range(1)). Not confirmed for every platform.

Thanks @Eclips4 for hints to finding the best reproduction. Thanks @trag1c for reporting the problem after accidentally running into it in a real-life use case.

I'm delighted to accept any help or guidance (cc @JelleZijlstra @carljm), as I want to author the patch to fix this issue and learn something in the process. Thanks!

CPython versions tested on:

3.13, 3.14, CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-128396: Fix a crash when inline comprehension has the same local variable as the outside scope #130235
• [3.13] gh-128396: Fix a crash when inline comprehension has the same … #130311

Footnotes

1. Const lists (from displays) are compiled into tuples under the hood before the loop begins, so [None] here is equivalent to (None,). ↩

[ZeroIntensity]

On a debug build, an assertion fails for all three reproducers:

python: Objects/frameobject.c:40: framelocalsproxy_getval: Assertion `PyCell_Check(value)' failed.

[ZeroIntensity]

@bswck Need any help here?

[carljm]

I'm very short on time at the moment, but @bswck if you have specific questions that you run into when looking into this, I'm happy to help try to answer them. (You can contact me on Discord or here.)

[bswck]

To reach the root cause, I started working on this by disabling assertions but keeping debug symbols and no optimizations:

./configure --with-{pydebug,trace-refs} CFLAGS="-DNDEBUG -O0 -g3" --prefix=/home/bswck/.local.

I'm posting this as these are my beginnings in C and I'm open to improvements of my workflow.

[bswck]

The segfault happens at refcount.h:275:17, in an #elif after #if defined(Py_GIL_DISABLED):

cpython/Include/refcount.h

Lines 274 to 276 in c357d69

	#elif SIZEOF_VOID_P > 4
	PY_UINT32_T cur_refcnt = op->ob_refcnt;
	if (((int32_t)cur_refcnt) < 0) {

So if I'm guessing correctly, there is a probability it won't happen in free-threaded builds?

[ZeroIntensity]

I doubt it. In general, crashes from macros and common operations (i.e., Py_INCREF in this case) are generally incidental, and not the actual cause of the crash. Here, an object is invalid, and that only shows up when something tries to increment its reference count.

[bswck]

Indeed! After running in the free-threaded build, the segfault also happens but in pyatomic_gcc.h:367:10.

cpython/Include/cpython/pyatomic_gcc.h

Lines 365 to 368 in c357d69

	static inline uint32_t
	_Py_atomic_load_uint32_relaxed(const uint32_t *obj)
	{ return __atomic_load_n(obj, __ATOMIC_RELAXED); }

[gaogaotiantian]

I can take a look at it. I'm familiar with both PEPs (seems like the assertion failure was written by me :)).