3.12: Python/bytecodes.c:2995: _PyEval_EvalFrameDefault: Assertion method != NULL failed` in boto3 test suite

[mgorny]

Crash report

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.12.0rc1+ (heads/3.12:09487a202f, Aug 25 2023, 15:57:40) [GCC 13.2.0]

What happened?

When CPython 3.12 is built with assertions enabled, the boto3 test suite crashes in tests/unit/test_session.py if it is run after a test from tests/functional/test_session.py. I can only reproduce this with Python 3.12, and I've tested as far as back as to 3.12.0b4. Pythons 3.11 and 3.10 work fine.

My reproducer so far:

git clone https://github.com/boto/boto3/
cd boto3
git checkout 1.28.34
pip install . pytest
python -m pytest -s tests/functional/test_session.py::TestSession::test_can_access_region_property tests/unit/test_session.py::TestSession::test_arguments_not_required

Error messages

pytest output

========================================================= test session starts =========================================================
platform linux -- Python 3.12.0rc1+, pytest-7.4.0, pluggy-1.2.0
rootdir: /tmp/boto3
configfile: pyproject.toml
collected 1302 items                                                                                                                  

tests/unit/test_boto3.py ........                                                                                               [  0%]
tests/unit/test_session.py Fatal Python error: Aborted

Current thread 0x00007f0e2e4aa740 (most recent call first):
  File "/tmp/boto3/boto3/session.py", line 132 in _setup_loader
  File "/tmp/boto3/boto3/session.py", line 90 in __init__
  File "/tmp/boto3/tests/unit/test_session.py", line 57 in test_arguments_not_required
  File "/home/mgorny/git/cpython/Lib/unittest/case.py", line 589 in _callTestMethod
  File "/home/mgorny/git/cpython/Lib/unittest/case.py", line 634 in run
  File "/home/mgorny/git/cpython/Lib/unittest/case.py", line 690 in __call__
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/_pytest/unittest.py", line 333 in runtest
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 80 in _multicall
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 112 in _hookexec
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 433 in __call__
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 341 in from_call
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 80 in _multicall
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 112 in _hookexec
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 433 in __call__
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/_pytest/main.py", line 349 in pytest_runtestloop
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 80 in _multicall
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 112 in _hookexec
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 433 in __call__
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/_pytest/main.py", line 324 in _main
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/_pytest/main.py", line 270 in wrap_session
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 80 in _multicall
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 112 in _hookexec
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 433 in __call__
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py", line 166 in main
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py", line 189 in console_main
  File "/tmp/boto3/.venv/lib/python3.12/site-packages/pytest/__main__.py", line 5 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main
Aborted (core dumped)

gdb backtrace

Core was generated by `python -m pytest tests/unit tests/functional'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44	pthread_kill.c: No such file or directory.
(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007f0e2e289e2f in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007f0e2e239cc2 in __GI_raise (sig=6) at ../sysdeps/posix/raise.c:26
#3  <signal handler called>
#4  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#5  0x00007f0e2e289e2f in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#6  0x00007f0e2e239cc2 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#7  0x00007f0e2e2224ed in __GI_abort () at abort.c:79
#8  0x00007f0e2e222415 in __assert_fail_base (fmt=0x7f0e2e39fb98 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=assertion@entry=0x565179f2a6bd "method != NULL", file=file@entry=0x565179f2a280 "Python/bytecodes.c", 
    line=line@entry=2995, function=function@entry=0x565179fa2af0 <__PRETTY_FUNCTION__.65> "_PyEval_EvalFrameDefault") at assert.c:92
#9  0x00007f0e2e2325e2 in __assert_fail (assertion=assertion@entry=0x565179f2a6bd "method != NULL", 
    file=file@entry=0x565179f2a280 "Python/bytecodes.c", line=line@entry=2995, 
    function=function@entry=0x565179fa2af0 <__PRETTY_FUNCTION__.65> "_PyEval_EvalFrameDefault") at assert.c:101
#10 0x0000565179c3dc24 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7f0e2e594720, throwflag=<optimized out>)
    at Python/bytecodes.c:2995
#11 0x0000565179cab64d in _PyObject_FastCallDictTstate (tstate=0x56517a2797b0 <_PyRuntime+459600>, callable=0x7f0e2c0ed300, 
    args=0x7ffd5b8b71a0, nargsf=1, kwargs=0x0) at Objects/call.c:133
#12 0x0000565179cab88a in _PyObject_Call_Prepend (tstate=tstate@entry=0x56517a2797b0 <_PyRuntime+459600>, 
    callable=callable@entry=0x7f0e2c0ed300, obj=obj@entry=0x7f0e297761e0, args=args@entry=0x56517a21be48 <_PyRuntime+76264>, 
    kwargs=kwargs@entry=0x0) at Objects/call.c:508
#13 0x0000565179d46e14 in slot_tp_init (self=0x7f0e297761e0, args=0x56517a21be48 <_PyRuntime+76264>, kwds=0x0)
    at Objects/typeobject.c:9007
#14 0x0000565179d402d8 in type_call (type=0x56517b652b10, args=0x56517a21be48 <_PyRuntime+76264>, kwds=0x0)
    at Objects/typeobject.c:1673
#15 0x0000565179ca86f0 in _PyObject_MakeTpCall (tstate=0x56517a2797b0 <_PyRuntime+459600>, callable=0x56517b652b10, 
    args=0x7f0e2e594650, nargs=0, keywords=<optimized out>) at Objects/call.c:240
#16 0x0000565179ca95e7 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, 
    callable=<optimized out>, tstate=<optimized out>) at ./Include/internal/pycore_call.h:90
#17 0x0000565179c3c0a0 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7f0e2e5945d8, throwflag=<optimized out>)
    at Python/bytecodes.c:2657
#18 0x0000565179cae087 in _PyObject_VectorcallTstate (kwnames=0x7f0e299dd000, nargsf=1, args=0x7f0e29aa9fe0, callable=0x7f0e2c65c900, 
    tstate=0x56517a2797b0 <_PyRuntime+459600>) at ./Include/internal/pycore_call.h:92
#19 method_vectorcall (method=method@entry=0x7f0e296ba7c0, args=args@entry=0x7f0e29aa9fe8, nargsf=<optimized out>, 
    kwnames=kwnames@entry=0x7f0e299dd000) at Objects/classobject.c:61
#20 0x0000565179caba8e in _PyVectorcall_Call (tstate=0x56517a2797b0 <_PyRuntime+459600>, func=0x565179caddd0 <method_vectorcall>, 
    callable=0x7f0e296ba7c0, tuple=<optimized out>, kwargs=<optimized out>) at Objects/call.c:283
#21 0x0000565179c308ee in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7f0e2e594410, throwflag=<optimized out>)
    at Python/bytecodes.c:3205
#22 0x0000565179cab6c1 in _PyObject_FastCallDictTstate (tstate=0x56517a2797b0 <_PyRuntime+459600>, callable=0x7f0e2c65cae0, 
    args=<optimized out>, nargsf=<optimized out>, kwargs=0x7f0e2bb08f40) at Objects/call.c:144
#23 0x0000565179cab88a in _PyObject_Call_Prepend (tstate=tstate@entry=0x56517a2797b0 <_PyRuntime+459600>, 
    callable=callable@entry=0x7f0e2c65cae0, obj=obj@entry=0x7f0e298818b0, args=args@entry=0x56517a21be48 <_PyRuntime+76264>, 
    kwargs=kwargs@entry=0x7f0e2bb08f40) at Objects/call.c:508
#24 0x0000565179d4b42d in slot_tp_call (self=0x7f0e298818b0, args=0x56517a21be48 <_PyRuntime+76264>, kwds=0x7f0e2bb08f40)
    at Objects/typeobject.c:8763
#25 0x0000565179ca86f0 in _PyObject_MakeTpCall (tstate=0x56517a2797b0 <_PyRuntime+459600>, callable=0x7f0e298818b0, 
    args=0x7f0e2e5943f0, nargs=0, keywords=<optimized out>) at Objects/call.c:240
#26 0x0000565179ca95e7 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, 
    callable=<optimized out>, tstate=<optimized out>) at ./Include/internal/pycore_call.h:90
#27 0x0000565179c3c0a0 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7f0e2e594380, throwflag=<optimized out>)
    at Python/bytecodes.c:2657
#28 0x0000565179cab6c1 in _PyObject_FastCallDictTstate (tstate=0x56517a2797b0 <_PyRuntime+459600>, callable=0x7f0e2d38f600, 
    args=<optimized out>, nargsf=<optimized out>, kwargs=0x7f0e2bb0a1c0) at Objects/call.c:144
#29 0x0000565179cab88a in _PyObject_Call_Prepend (tstate=tstate@entry=0x56517a2797b0 <_PyRuntime+459600>, 
    callable=callable@entry=0x7f0e2d38f600, obj=obj@entry=0x7f0e2c530d60, args=args@entry=0x56517a21be48 <_PyRuntime+76264>, 
    kwargs=kwargs@entry=0x7f0e2bb0a1c0) at Objects/call.c:508
#30 0x0000565179d4b42d in slot_tp_call (self=0x7f0e2c530d60, args=0x56517a21be48 <_PyRuntime+76264>, kwds=0x7f0e2bb0a1c0)
    at Objects/typeobject.c:8763
#31 0x0000565179cabe69 in PyObject_Call () at Objects/call.c:376
#32 0x0000565179c308ee in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7f0e2e594028, throwflag=<optimized out>)
    at Python/bytecodes.c:3205
#33 0x0000565179cab6c1 in _PyObject_FastCallDictTstate (tstate=0x56517a2797b0 <_PyRuntime+459600>, callable=0x7f0e2d38f600, 
    args=<optimized out>, nargsf=<optimized out>, kwargs=0x7f0e29822e00) at Objects/call.c:144
#34 0x0000565179cab88a in _PyObject_Call_Prepend (tstate=tstate@entry=0x56517a2797b0 <_PyRuntime+459600>, 
    callable=callable@entry=0x7f0e2d38f600, obj=obj@entry=0x7f0e2c530ef0, args=args@entry=0x56517a21be48 <_PyRuntime+76264>, 
    kwargs=kwargs@entry=0x7f0e29822e00) at Objects/call.c:508
#35 0x0000565179d4b42d in slot_tp_call (self=0x7f0e2c530ef0, args=0x56517a21be48 <_PyRuntime+76264>, kwds=0x7f0e29822e00)
    at Objects/typeobject.c:8763
#36 0x0000565179ca86f0 in _PyObject_MakeTpCall (tstate=0x56517a2797b0 <_PyRuntime+459600>, callable=0x7f0e2c530ef0, 
    args=0x7f0e2e593a58, nargs=0, keywords=<optimized out>) at Objects/call.c:240
#37 0x0000565179ca95e7 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, 
--Type <RET> for more, q to quit, c to continue without paging--c
    callable=<optimized out>, tstate=<optimized out>) at ./Include/internal/pycore_call.h:90
#38 0x0000565179c3c0a0 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7f0e2e5939d8, throwflag=<optimized out>)
    at Python/bytecodes.c:2657
#39 0x0000565179cab6c1 in _PyObject_FastCallDictTstate (tstate=0x56517a2797b0 <_PyRuntime+459600>, callable=0x7f0e2d38f600, 
    args=<optimized out>, nargsf=<optimized out>, kwargs=0x7f0e298229c0) at Objects/call.c:144
#40 0x0000565179cab88a in _PyObject_Call_Prepend (tstate=tstate@entry=0x56517a2797b0 <_PyRuntime+459600>, 
    callable=callable@entry=0x7f0e2d38f600, obj=obj@entry=0x7f0e2c530fe0, args=args@entry=0x56517a21be48 <_PyRuntime+76264>, 
    kwargs=kwargs@entry=0x7f0e298229c0) at Objects/call.c:508
#41 0x0000565179d4b42d in slot_tp_call (self=0x7f0e2c530fe0, args=0x56517a21be48 <_PyRuntime+76264>, kwds=0x7f0e298229c0)
    at Objects/typeobject.c:8763
#42 0x0000565179ca86f0 in _PyObject_MakeTpCall (tstate=0x56517a2797b0 <_PyRuntime+459600>, callable=0x7f0e2c530fe0, 
    args=0x7f0e2e593778, nargs=0, keywords=<optimized out>) at Objects/call.c:240
#43 0x0000565179ca95e7 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, 
    callable=<optimized out>, tstate=<optimized out>) at ./Include/internal/pycore_call.h:90
#44 0x0000565179c3c0a0 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7f0e2e593710, throwflag=<optimized out>)
    at Python/bytecodes.c:2657
#45 0x0000565179cab6c1 in _PyObject_FastCallDictTstate (tstate=0x56517a2797b0 <_PyRuntime+459600>, callable=0x7f0e2d38f600, 
    args=<optimized out>, nargsf=<optimized out>, kwargs=0x7f0e2c400040) at Objects/call.c:144
#46 0x0000565179cab88a in _PyObject_Call_Prepend (tstate=tstate@entry=0x56517a2797b0 <_PyRuntime+459600>, 
    callable=callable@entry=0x7f0e2d38f600, obj=obj@entry=0x7f0e2c530220, args=args@entry=0x56517a21be48 <_PyRuntime+76264>, 
    kwargs=kwargs@entry=0x7f0e2c400040) at Objects/call.c:508
#47 0x0000565179d4b42d in slot_tp_call (self=0x7f0e2c530220, args=0x56517a21be48 <_PyRuntime+76264>, kwds=0x7f0e2c400040)
    at Objects/typeobject.c:8763
#48 0x0000565179ca86f0 in _PyObject_MakeTpCall (tstate=0x56517a2797b0 <_PyRuntime+459600>, callable=0x7f0e2c530220, 
    args=0x7f0e2e593358, nargs=0, keywords=<optimized out>) at Objects/call.c:240
#49 0x0000565179ca95e7 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, 
    callable=<optimized out>, tstate=<optimized out>) at ./Include/internal/pycore_call.h:90
#50 0x0000565179c3c0a0 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x56517a2797b0 <_PyRuntime+459600>, frame=0x7f0e2e5932a8, 
    frame@entry=0x7f0e2e5931b8, throwflag=throwflag@entry=0) at Python/bytecodes.c:2657
#51 0x0000565179df78d1 in _PyEval_EvalFrame (throwflag=0, frame=0x7f0e2e5931b8, tstate=0x56517a2797b0 <_PyRuntime+459600>)
    at ./Include/internal/pycore_ceval.h:88
#52 _PyEval_Vector (args=0x0, argcount=0, kwnames=0x0, locals=0x7f0e2e1f9cc0, func=0x7f0e2e1ddb20, 
    tstate=0x56517a2797b0 <_PyRuntime+459600>) at Python/ceval.c:1675
#53 PyEval_EvalCode (co=co@entry=0x7f0e2c6cb440, globals=globals@entry=0x7f0e2e1f9cc0, locals=locals@entry=0x7f0e2e1f9cc0)
    at Python/ceval.c:570
#54 0x0000565179df2920 in builtin_exec_impl (module=<optimized out>, closure=<optimized out>, locals=0x7f0e2e1f9cc0, 
    globals=0x7f0e2e1f9cc0, source=0x7f0e2c6cb440) at Python/bltinmodule.c:1096
#55 builtin_exec (module=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>)
    at Python/clinic/bltinmodule.c.h:586
#56 0x0000565179d18d6c in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x7f0e2e195c60, args=0x7f0e2e593180, nargsf=<optimized out>, 
    kwnames=0x0) at Objects/methodobject.c:438
#57 0x0000565179ca9573 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775810, args=0x7f0e2e593180, 
    callable=0x7f0e2e195c60, tstate=0x56517a2797b0 <_PyRuntime+459600>) at ./Include/internal/pycore_call.h:92
#58 PyObject_Vectorcall (callable=0x7f0e2e195c60, args=0x7f0e2e593180, nargsf=9223372036854775810, kwnames=0x0) at Objects/call.c:325
#59 0x0000565179c3c0a0 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7f0e2e5930d8, throwflag=<optimized out>)
    at Python/bytecodes.c:2657
#60 0x0000565179e95719 in pymain_run_module (modname=<optimized out>, set_argv0=set_argv0@entry=1) at Modules/main.c:300
#61 0x0000565179e95f6a in pymain_run_python (exitcode=0x7ffd5b8b8c60) at Modules/main.c:604
#62 0x0000565179e9695a in Py_RunMain () at Modules/main.c:689
#63 pymain_main (args=0x7ffd5b8b8c20) at Modules/main.c:719
#64 Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:743
#65 0x00007f0e2e223f0a in __libc_start_call_main (main=main@entry=0x565179c2f800 <main>, argc=argc@entry=5, 
    argv=argv@entry=0x7ffd5b8b8da8) at ../sysdeps/nptl/libc_start_call_main.h:58
#66 0x00007f0e2e223fc5 in __libc_start_main_impl (main=0x565179c2f800 <main>, argc=5, argv=0x7ffd5b8b8da8, init=<optimized out>, 
    fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffd5b8b8d98) at ../csu/libc-start.c:360
#67 0x0000565179c403b1 in _start ()

Linked PRs

• [3.12] gh-108487: Fix crash due to incorrect assert #108508
• [3.12] gh-108487: Change assert that should've been DEOPT_IF #108509
• gh-108487: Move assert(self != NULL) down beyond DEOPT_IF() #108510

[mgorny]

Oh, and my first suspicion was that pytest is doing something funny because both test files have the same filename but renaming one of them didn't change anything.

[mgorny]

Well, I managed to reduce it some. I can reproduce the failure using the following script:

import boto3.session
from unittest import mock


boto3.session.Session()
boto3.session.Session()

bc_session_patch = mock.patch('botocore.session.Session')
bc_session_patch.start()
boto3.session.Session()

I'm going to try reducing it even further.

[mgorny]

As a next step, I've managed to reproduce it using botocore alone:

import os.path
from unittest import mock

import botocore.session


def Session():
    session = botocore.session.get_session()
    loader = session.get_component('data_loader')
    loader.search_paths.append('xxx')


Session()
Session()

bc_session_patch = mock.patch('botocore.session.Session')
bc_session_patch.start()
Session()

[mgorny]

Ok, I think that's how far as I can get it.

repro_core.py:

class CoreSession:
    search_paths = []

repro.py:

from unittest import mock

import repro_core


def Session():
    session = repro_core.CoreSession()
    session.search_paths.append(1)


Session()
Session()

bc_session_patch = mock.patch('repro_core.CoreSession')
bc_session_patch.start()
Session()

[mgorny]

I've been able to bisect this to 616aec1:

commit 616aec1ff148ba4570aa2d4b8ea420c715c206e4 (HEAD)
Author:     Guido van Rossum <guido@python.org>
AuthorDate: 2023-02-08 20:40:10 +0100
Commit:     GitHub <noreply@github.com>
CommitDate: 2023-02-08 20:40:10 +0100

    gh-98831: Modernize CALL and family (#101508)
    
    Includes a slight improvement to `DECREF_INPUTS()`.

CC @gvanrossum

[gvanrossum]

Thanks for the report. Will try to figure out where we are going wrong. Hopefully it’s not too dramatic.

[gvanrossum]

Let’s call it a release blocker at least until I’ve had time to look into it more.

[gvanrossum]

The failing assert shouldn't exist -- this condition should result in a deoptimization, and will, two lines below. I will prepare PRs for main and 3.12 later.