Parser/lexer/lexer.c:1218: int tok_get_normal_mode(struct tok_state *, tokenizer_mode *, struct token *): Assertion `current_tok->curly_bracket_depth >= 0' failed.

[alex]

Crash report

What happened?

~/p/cpython ❯❯❯ ./python.exe -c "import ast; ast.literal_eval(\"F'{[F'{:'}[F'{:'}]]]\")"
Assertion failed: (current_tok->curly_bracket_depth >= 0), function tok_get_normal_mode, file lexer.c, line 1218.
fish: Job 1, './python.exe -c "import ast; as…' terminated by signal SIGABRT (Abort)

Full ASAN stack:

==1327==ERROR: AddressSanitizer: ABRT on unknown address 0x05390000052f (pc 0x7add17cb300b bp 0x7add17e28588 sp 0x7ffcd862dbf0 T0)
SCARINESS: 10 (signal)
    #0 0x7add17cb300b in raise /build/glibc-SzIz7B/glibc-2.31/sysdeps/unix/sysv/linux/raise.c:51:1
    #1 0x7add17c92858 in abort /build/glibc-SzIz7B/glibc-2.31/stdlib/abort.c:79:7
    #2 0x7add17c92728 in __assert_fail_base /build/glibc-SzIz7B/glibc-2.31/assert/assert.c:92:3
    #3 0x7add17ca3fd5 in __assert_fail /build/glibc-SzIz7B/glibc-2.31/assert/assert.c:101:3
    #4 0x5ad0da7419ae in tok_get_normal_mode cpython3/Parser/lexer/lexer.c:1218:21
    #5 0x5ad0da732ea1 in tok_get cpython3/Parser/lexer/lexer.c:1483:16
    #6 0x5ad0da732ea1 in _PyTokenizer_Get cpython3/Parser/lexer/lexer.c:1492:18
    #7 0x5ad0da67f20e in _PyPegen_tokenize_full_source_to_check_for_errors cpython3/Parser/pegen_errors.c:178:17
    #8 0x5ad0da67b0ec in _PyPegen_run_parser cpython3/Parser/pegen.c:888:9
    #9 0x5ad0da67ba95 in _PyPegen_run_parser_from_string cpython3/Parser/pegen.c:992:14
    #10 0x5ad0da25b0d5 in Py_CompileStringObject cpython3/Python/pythonrun.c:1435:11
    #11 0x5ad0da085182 in builtin_compile_impl cpython3/Python/bltinmodule.c:878:14
    #12 0x5ad0da085182 in builtin_compile cpython3/Python/clinic/bltinmodule.c.h:361:20
    #13 0x5ad0da57cef2 in cfunction_vectorcall_FASTCALL_KEYWORDS cpython3/Objects/methodobject.c:441:24
    #14 0x5ad0d9d9c798 in _PyObject_VectorcallTstate cpython3/Include/internal/pycore_call.h:167:11
    #15 0x5ad0da0b9479 in _PyEval_EvalFrameDefault cpython3/Python/generated_cases.c.h:1647:31
    #16 0x5ad0d9d9c798 in _PyObject_VectorcallTstate cpython3/Include/internal/pycore_call.h:167:11
    #17 0x5ad0d9d9fd65 in PyObject_CallOneArg cpython3/Objects/call.c:395:12
    #18 0x5ad0d9d9bc37 in fuzz_ast_literal_eval cpython3/Modules/_xxtestfuzz/fuzzer.c:426:25
    #19 0x5ad0d9d9bc37 in _run_fuzz cpython3/Modules/_xxtestfuzz/fuzzer.c:570:14
    #20 0x5ad0d9d9bc37 in LLVMFuzzerTestOneInput cpython3/Modules/_xxtestfuzz/fuzzer.c:697:11
    #21 0x5ad0d9c4e040 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:614:13
    #22 0x5ad0d9c387d4 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:327:6
    #23 0x5ad0d9c3e26a in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:862:9
    #24 0x5ad0d9c6a662 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
    #25 0x7add17c94082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/libc-start.c:308:16
    #26 0x5ad0d9c2f2ad in _start

Regression range in ece20db...6be7aee

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Output from running 'python -VV' on the command line:

No response

Linked PRs

• gh-122026: Fix identification of mismatched parentheses inside f-strings #122028
• [3.13] gh-122026: Fix identification of mismatched parentheses inside f-strings (GH-122028) #122041
• [3.12] gh-122026: Fix identification of mismatched parentheses inside f-strings (GH-122028) #122062

[alex]

cc: @pablogsal I think this may be related to the PEP 701 work

[pablogsal]

Regression range in ece20db...6be7aee

These are quite old commits but the stack trace is using the new tokeniser structure. What version of Python was used to trigger this bug?

Also I am currently unable to reproduce it using the current main:

./python.exe -c "import ast; ast.literal_eval(\"F'{[F'{:'}[F'{:'}]]]\")"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import ast; ast.literal_eval("F'{[F'{:'}[F'{:'}]]]")
                ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pgalindo3/github/python/main/Lib/ast.py", line 67, in literal_eval
    node_or_string = parse(node_or_string.lstrip(" \t"), mode='eval')
  File "/Users/pgalindo3/github/python/main/Lib/ast.py", line 53, in parse
    return compile(source, filename, mode, flags,
                   _feature_version=feature_version, optimize=optimize)
  File "<unknown>", line 1
    F'{[F'{:'}[F'{:'}]]]
           ^
SyntaxError: f-string: valid expression required before ':'

[pablogsal]

Ah, this only happens in debug mode (because it's an assert). Got it

[alex]

Yes, sorry, should have said that. ./configure --with-pydebug --with-address-sanitizer --with-assertions && make -j is what I build with for verifying fuzzing crashers.