Python/compile.c:7482: int compute_code_flags(struct compiler *): Assertion `IS_TOP_LEVEL_AWAIT(c) || _PyST_IsFunctionLike(ste)' failed

[alex]

Crash report

What happened?

~/p/cpython ❯❯❯ ./python.exe -c 'compile("assert await u", "", "exec", optimize=2)'
Assertion failed: (IS_TOP_LEVEL_AWAIT(c) || _PyST_IsFunctionLike(ste)), function compute_code_flags, file compile.c, line 7482.
fish: Job 1, './python.exe -c 'compile("asser…' terminated by signal SIGABRT (Abort)

Full stacktrace:

fuzz_pycompile: Python/compile.c:7482: int compute_code_flags(struct compiler *): Assertion `IS_TOP_LEVEL_AWAIT(c) \|\| _PyST_IsFunctionLike(ste)' failed.
--
  | ==24717== ERROR: libFuzzer: deadly signal
  | #0 0x59033612cf71 in __sanitizer_print_stack_trace /src/llvm-project/compiler-rt/lib/asan/asan_stack.cpp:87:3
  | #1 0x59033602fd98 in fuzzer::PrintStackTrace() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerUtil.cpp:210:5
  | #2 0x590336012ac3 in fuzzer::Fuzzer::CrashCallback() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:231:3
  | #3 0x79353882e41f in libpthread.so.0
  | #4 0x79353862800a in __libc_signal_restore_set /build/glibc-SzIz7B/glibc-2.31/sysdeps/unix/sysv/linux/internal-signals.h:86:3
  | #5 0x79353862800a in raise /build/glibc-SzIz7B/glibc-2.31/sysdeps/unix/sysv/linux/raise.c:48:3
  | #6 0x793538607858 in abort /build/glibc-SzIz7B/glibc-2.31/stdlib/abort.c:79:7
  | #7 0x793538607728 in __assert_fail_base /build/glibc-SzIz7B/glibc-2.31/assert/assert.c:92:3
  | #8 0x793538618fd5 in __assert_fail /build/glibc-SzIz7B/glibc-2.31/assert/assert.c:101:3
  | #9 0x5903364b52da in compute_code_flags cpython3/Python/compile.c:7482:9
  | #10 0x5903364b52da in optimize_and_assemble cpython3/Python/compile.c:7578:22
  | #11 0x5903364af709 in compiler_mod cpython3/Python/compile.c:1597:10
  | #12 0x5903364af709 in _PyAST_Compile cpython3/Python/compile.c:422:24
  | #13 0x59033660c8a0 in Py_CompileStringObject cpython3/Python/pythonrun.c:1450:10
  | #14 0x59033660c994 in Py_CompileStringExFlags cpython3/Python/pythonrun.c:1463:10
  | #15 0x590336161d2e in fuzz_pycompile cpython3/Modules/_xxtestfuzz/fuzzer.c:551:24
  | #16 0x590336161d2e in _run_fuzz cpython3/Modules/_xxtestfuzz/fuzzer.c:570:14
  | #17 0x590336161d2e in LLVMFuzzerTestOneInput cpython3/Modules/_xxtestfuzz/fuzzer.c:711:11
  | #18 0x590336013fe0 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:614:13
  | #19 0x590335ffe774 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:327:6
  | #20 0x59033600420a in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:862:9
  | #21 0x590336030602 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
  | #22 0x793538609082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/libc-start.c:308:16
  | #23 0x590335ff524d in _start
  |  

This regressed somewhere in f621618...690b935

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

No response

Linked PRs

• gh-121637: Syntax error for optimized-away incorrect await #121656

[alex]

cc: @iritkatriel -- it looks like you've got a few commits in this file in that range, though I don't see one that's obviously relevant.

[JelleZijlstra]

Feels related to #121297.

[sobolevn]

I will add both asyncio and interpreter-core, because both are related.

[JelleZijlstra]

Irit is on vacation, I'll look into this over the next few days.

[JelleZijlstra]

The problem is arguably this:

% python -c 'assert await 42'   
  File "<string>", line 1
SyntaxError: 'await' outside function
% python -O -c 'assert await 42'
%

It used to be that we didn't raise a SyntaxError if there is an await but it is optimized away, and the code would work fine.

Now, however, the symtable sets ste_coroutine when it sees the await, but it still doesn't raise an error about the await, leading to the crash reported here.

I can see two solutions:

• Make the code setting ste_coroutine aware of -O removing asserts, either by reverting gh-121272: compiler no longer modifies the symtable, ste_coroutine always set during symtable construction #121297 or by adding an awareness of asserts into symtable.c. The second suboption would be quite complex, and the first reintroduces the layering violation that gh-121272: compiler no longer modifies the symtable, ste_coroutine always set during symtable construction #121297 tried to fix.
• Make it so it's still a SyntaxError to await outside of a coroutine if the await is optimized away.

The latter seems like the more intuitive behavior for users.

It's also consistent with other cases where optimized-away code leads to a compile error:

% python -O
Python 3.12.1 (main, Apr 22 2024, 00:22:15) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> def f():
...     assert (x := 3)
...     global x
... 
  File "<stdin>", line 3
SyntaxError: name 'x' is assigned to before global declaration

So I'll send a PR to make python -O -c 'assert await 1' a SyntaxError.

[alex]

Always doing a syntax error, even if the assert is optimized out, feels like the right behavior to me.

[JelleZijlstra]

I realized there's a few more similar cases (essentially, every place we call compiler_error in compile.c while inside an expression):

% python -c 'assert (yield)'
  File "<string>", line 1
SyntaxError: 'yield' outside function
% python -O -c 'assert (yield)'
% python -O -c 'assert (__debug__ := True)'
% python -c 'assert (__debug__ := True)' 
  File "<string>", line 1
SyntaxError: cannot assign to __debug__
% python -O -c 'assert (__debug__ := True)'
% python -c 'assert [x async for x in ()]' 
  File "<string>", line 1
SyntaxError: asynchronous comprehension outside of an asynchronous function
% python -O -c 'assert [x async for x in ()]'

The latter also crashes on current main:

% ./python.exe -O -c 'assert [x async for x in ()]'
Assertion failed: (IS_TOP_LEVEL_AWAIT(c) || _PyST_IsFunctionLike(ste)), function compute_code_flags, file compile.c, line 7482.
zsh: abort      ./python.exe -O -c 'assert [x async for x in ()]'

[JelleZijlstra]

The linked PR makes symtable.c detect incorrect await and async comprehensions, fixing this issue.

I didn't bother detecting other compiler errors (e.g., invalid yields) because they don't cause similar crashes. I wouldn't be surprised though if there's a crash lurking somewhere with e.g. assert (yield) being optimized away, so maybe if you point your fuzzer in that direction it will find something :)