PPC64LE Fedora Stable Clang 3.x fails test_gdb

[JelleZijlstra]

Buildbot https://buildbot.python.org/all/#/builders/435 has been failing test_gdb with errors like these:

======================================================================
FAIL: test_pycfunction (test.test_gdb.PyBtTests.test_pycfunction) [_testcapi.meth_varargs]
Verify that "py-bt" displays invocations of PyCFunction instances
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-ppc64le.clang/build/Lib/test/test_gdb.py", line 944, in test_pycfunction
    self.assertIn(f'<built-in method {func_name}', gdb_output)
AssertionError: '<built-in method meth_varargs' not found in 'Breakpoint 1 (meth_varargs) pending.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib64/libthread_db.so.1".\n\nBreakpoint 1, meth_varargs (self=<module at remote 0x7fffea347230>, args=()) at ./Modules/_testcapimodule.c:2307\n2307\t    return Py_BuildValue("NO", _null_to_none(self), args);\n#0  meth_varargs (self=<module at remote 0x7fffea347230>, args=()) at ./Modules/_testcapimodule.c:2307\n#1  PyModuleDef_Type ()\nBacktrace stopped: frame did not save the PC\nUnable to locate python frame\n'

It last succeeded on commit 373bca0 and first failed on fcd5fb4, suggesting that the culprit is one of these commits:

• fcd5fb4 Comprehension inlining: Bug if comprehension contains a lambda #104357: fix inlined comprehensions that close over iteration var (gh-104357: fix inlined comprehensions that close over iteration var #104368)
• 94f30c7 [pathlib] Option so that OSError does not block glob in pathlib library #90208: Suppress OSError exceptions from pathlib.Path.glob() (GH-90208: Suppress OSError exceptions from pathlib.Path.glob() #104141)

Neither looks likely to affect test_gdb though.

Linked PRs

• gh-104736: Fix test_gdb tests on ppc64le with clang #109360
• [3.12] gh-104736: Fix test_gdb tests on ppc64le with clang (GH-109360) #109361
• [3.11] gh-104736: Fix test_gdb tests on ppc64le with clang (GH-109360) #109362

[encukou]

It looks like GDB can't extract the C stack here. I SSHd to the buildbot and tried:

$ gdb --args ./python -c 'import _testcapi; _testcapi.meth_varargs()'
[...]
Reading symbols from ./python...

(gdb) b meth_varargs
Function "meth_varargs" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (meth_varargs) pending.

(gdb) r
[...]
Breakpoint 1, meth_varargs (self=<module at remote 0x7ffff6f77d70>, args=())
    at ./Modules/_testcapimodule.c:2307
2307	    return Py_BuildValue("NO", _null_to_none(self), args);

(gdb) bt
#0  meth_varargs (self=<module at remote 0x7ffff6f77d70>, args=()) at ./Modules/_testcapimodule.c:2307
#1  0x00007ffff7cf7db0 in PyModuleDef_Type () from /home/pviktori/cpython/libpython3.12d.so.1.0
Backtrace stopped: frame did not save the PC

The stack looks corrupted: PyModuleDef_Type is data, not a function.
This looks like “miscommunication” between clang and gdb (bad debug info?)

The test seems very sensitive to platform/compiler, e.g.:

if 'Clang' in platform.python_compiler() and sys.platform == 'darwin':
    raise unittest.SkipTest("test_gdb doesn't work correctly when python is"
                            " built with LLVM clang")

Note that the “installed” buildbots are green. (AFAIK these test Python after make install.) But when I make install, GDB still fails to extract the backtrace on the installed Python.

ping @vstinner, the PEP-11 ppc64le/clang maintainer

[encukou]

See #65462 (comment):

Unless someone has a better idea, I suggest we just disable test_gdb on OS X or, somewhat more precisely, when Python has been compiled with LLVM clang.

Doesn't clang always use LLVM?
The skip is now 'Clang' in platform.python_compiler() and sys.platform == 'darwin', should it just be 'Clang' in platform.python_compiler()?

[stratakis]

The failure started after the update of Clang to 16.0.0 from 15.0.7.

[thesamesam]

See #65462 (comment):

Unless someone has a better idea, I suggest we just disable test_gdb on OS X or, somewhat more precisely, when Python has been compiled with LLVM clang.

Doesn't clang always use LLVM? The skip is now 'Clang' in platform.python_compiler() and sys.platform == 'darwin', should it just be 'Clang' in platform.python_compiler()?

I took it to mean "not Apple's Clang" which is distinct (has its own versioning and is essentially a fork). I might be wrong here though.

[vstinner]

Maybe tets_gdb can skip a test if "frame did not save the PC" is found in the output. We already have many skips like that.

[carljm]

It looks like multiple Clang buildbots are still failing test_gdb.

[vstinner]

I closed issue #109053 as a duplicate of this issue.

[vstinner]

I did tests on two ppc64le machines. In short, test_gdb only failed on ppc64le if Python is built with clang version 16.0.6 (clang -Og). Well, that's what we saw on buildbots. But I preferred to confirm manually. I built Python manually and ran test_gdb manually with -u all to no skip any test.

• Machine 1:

  • Fedora 38
  • Linux xxx 6.2.8-200.fc37.ppc64le #1 SMP Wed Mar 22 18:34:43 UTC 2023 ppc64le ppc64le ppc64le GNU/Linux
  • gcc (GCC) 13.2.1 20230728 (Red Hat 13.2.1-1)
  • clang version 16.0.6 (Fedora 16.0.6-2.fc38)
  • GNU gdb (GDB) Fedora Linux 13.2-3.fc38

• Machine 2:

  • Fedora 37
  • Linux localhost.localdomain 5.8.17-200.fc32.ppc64le #1 SMP Thu Oct 29 18:00:08 UTC 2020 ppc64le GNU/Linux
  • gcc (GCC) 12.3.1 2023050 (Red Hat 12.3.1-1)
  • clang version 15.0.7 (Fedora 15.0.7-2.fc37)
  • GNU gdb (GDB) Fedora Linux 13.2-2.fc37

GCC

Test: ./configure --with-pydebug && make && ./python -m test test_gdb -u all -v.

Build built with: gcc -Og.

• Machine 1: test_gdb: SUCCESS

  • Only 1 test was skipped: test_print_after_up(): Unable to read information on python frame found in gdb output

• Machine 2: test_gdb: SUCCESS

  • Only 1 test was skipped: test_print_after_up(): Unable to read information on python frame found in gdb output"
  • Total tests: run=46 skipped=1

LLVM clang

Test: ./configure --with-pydebug CC=clang && make && ./python -m test test_gdb -u all -v.

Build built with: clang -Og.

• Machine 1: test_gdb: FAILURE

  • Failed tests: test_pycfunction() [_testcapi.meth_varargs] and test_pycfunction [_testcapi.meth_fastcall_keywords] failed.
  • On 46 tests, 39 were skipped: (unable to read python frame information) or Unable to read information on python frame found in gdb output.
  • Total tests: run=46 failures=2 skipped=39

• Machine 2: test_gdb: SUCCESS

  • Only 5 tests were run in fact, 41 have been skipped: (unable to read python frame information) or Unable to read information on python frame found in gdb output
  • Total tests: run=46 skipped=41

[vstinner]

Reproduce _testcapi.meth_varargs() issue.

script.py:

import _testcapi
def foo():
    _testcapi.meth_varargs()
def bar():
    foo()
bar()

cmds:

file ./python
source Tools/gdb/libpython.py
set breakpoint pending yes
break meth_varargs
run script.py
where
py-bt

gdb output on ppc64le with clang -Og:

$ gdb -x cmds
GNU gdb (GDB) Fedora Linux 13.2-3.fc38

Function "meth_varargs" not defined.
Breakpoint 1 (meth_varargs) pending.

Breakpoint 1, meth_varargs (self=<module at remote 0x7fffea3d23f0>, args=()) at ./Modules/_testcapimodule.c:2060
2060	    return Py_BuildValue("NO", _null_to_none(self), args);

#0  meth_varargs (self=<module at remote 0x7fffea3d23f0>, args=()) at ./Modules/_testcapimodule.c:2060
#1  0x0000000010625d78 in PyModuleDef_Type ()
Backtrace stopped: frame did not save the PC

Unable to locate python frame

The important message here is Backtrace stopped: frame did not save the PC. The gdb traceback is really just 2 frames and the 2nd frame PyModuleDef_Type () is wrong :-(

Comparison on x86-64, Python built with gcc -Og:

vstinner@mona$ gdb -x cmds
GNU gdb (GDB) Fedora Linux 13.2-3.fc38

Function "meth_varargs" not defined.
Breakpoint 1 (meth_varargs) pending.

Breakpoint 1, meth_varargs (self=self@entry=<module at remote 0x7fffea4c29f0>, args=args@entry=()) at ./Modules/_testcapimodule.c:2059
2059	{


#0  meth_varargs (self=self@entry=<module at remote 0x7fffea4c29f0>, args=args@entry=()) at ./Modules/_testcapimodule.c:2059
#1  0x00000000004f2dcc in cfunction_call (func=func@entry=<built-in method meth_varargs of module object at remote 0x7fffea4c29f0>, args=args@entry=(), 
    kwargs=kwargs@entry=0x0) at Objects/methodobject.c:551
#2  0x00000000004ab06f in _PyObject_MakeTpCall (tstate=tstate@entry=0x980358 <_PyRuntime+508728>, 
    callable=callable@entry=<built-in method meth_varargs of module object at remote 0x7fffea4c29f0>, args=args@entry=0x7ffff7fb9128, nargs=<optimized out>, 
    keywords=keywords@entry=0x0) at Objects/call.c:242
#3  0x00000000004ab282 in _PyObject_VectorcallTstate (tstate=0x980358 <_PyRuntime+508728>, 
    callable=callable@entry=<built-in method meth_varargs of module object at remote 0x7fffea4c29f0>, args=args@entry=0x7ffff7fb9128, nargsf=<optimized out>, 
    kwnames=kwnames@entry=0x0) at ./Include/internal/pycore_call.h:185
#4  0x00000000004ab2d0 in PyObject_Vectorcall (callable=callable@entry=<built-in method meth_varargs of module object at remote 0x7fffea4c29f0>, 
    args=args@entry=0x7ffff7fb9128, nargsf=<optimized out>, kwnames=kwnames@entry=0x0) at Objects/call.c:327
#5  0x000000000059e417 in _PyEval_EvalFrameDefault (tstate=0x980358 <_PyRuntime+508728>, frame=0x7ffff7fb90d0, throwflag=0) at Python/generated_cases.c.h:3773
(...)
#20 0x000000000041b78f in main (argc=<optimized out>, argv=<optimized out>) at ./Programs/python.c:15


Traceback (most recent call first):
  <built-in method meth_varargs of module object at remote 0x7fffea4c29f0>
  File "/home/vstinner/python/main/script.py", line 3, in foo
    _testcapi.meth_varargs()
  File "/home/vstinner/python/main/script.py", line 5, in bar
    foo()
  File "/home/vstinner/python/main/script.py", line 6, in <module>
    bar()

• gcc (GCC) 13.2.1 20230728 (Red Hat 13.2.1-1)
• GNU gdb (GDB) Fedora Linux 13.2-3.fc38

On x86-64 with gcc, I have 20 frames up to main() and the 2nd frame is cfunction_call() as expected.

[vstinner]

script2.py:

import _testcapi
def foo():
    _testcapi.meth_fastcall_keywords()
def bar():
    foo()
bar()

cmds2:

file ./python
source Tools/gdb/libpython.py
set breakpoint pending yes
break meth_fastcall_keywords
run script2.py
where
py-bt

gdb output on ppc64 with clang -Og:

$ gdb -x cmds2
GNU gdb (GDB) Fedora Linux 13.2-3.fc38

Function "meth_fastcall_keywords" not defined.
Breakpoint 1 (meth_fastcall_keywords) pending.

Breakpoint 1, meth_fastcall_keywords (self=<module at remote 0x7fffea3d23f0>, args=0x7fffea410128, nargs=0, kwargs=0x0) at ./Modules/_testcapimodule.c:2107
2107	    PyObject *pyargs = _fastcall_to_tuple(args, nargs);


#0  meth_fastcall_keywords (self=<module at remote 0x7fffea3d23f0>, args=0x7fffea410128, nargs=0, kwargs=0x0) at ./Modules/_testcapimodule.c:2107
#1  0x0000000000000000 in ?? ()


Unable to locate python frame

Here, I think that the important part is #1 0x0000000000000000 in ?? (). When gdb says that, usually it's likely that the traceback is truncated or wrong.

---

For comparison, gdb on x86-64 on Python built with gcc -Og:

vstinner@mona$ gdb -x cmds2
GNU gdb (GDB) Fedora Linux 13.2-3.fc38

Function "meth_fastcall_keywords" not defined.
Breakpoint 1 (meth_fastcall_keywords) pending.

Breakpoint 1, meth_fastcall_keywords (self=<module at remote 0x7fffea4c2a50>, args=args@entry=0x7ffff7fb9128, nargs=nargs@entry=0, kwargs=kwargs@entry=0x0)
    at ./Modules/_testcapimodule.c:2106
2106	{


#0  meth_fastcall_keywords (self=<module at remote 0x7fffea4c2a50>, args=args@entry=0x7ffff7fb9128, nargs=nargs@entry=0, kwargs=kwargs@entry=0x0)
    at ./Modules/_testcapimodule.c:2106
#1  0x00000000004f259a in cfunction_vectorcall_FASTCALL_KEYWORDS (func=<built-in method meth_fastcall_keywords of module object at remote 0x7fffea4c2a50>, 
    args=0x7ffff7fb9128, nargsf=<optimized out>, kwnames=0x0) at Objects/methodobject.c:441
#2  0x00000000004ab208 in _PyObject_VectorcallTstate (tstate=0x980358 <_PyRuntime+508728>, 
    callable=callable@entry=<built-in method meth_fastcall_keywords of module object at remote 0x7fffea4c2a50>, args=args@entry=0x7ffff7fb9128, nargsf=9223372036854775808, 
    kwnames=kwnames@entry=0x0) at ./Include/internal/pycore_call.h:187
#3  0x00000000004ab2d0 in PyObject_Vectorcall (callable=callable@entry=<built-in method meth_fastcall_keywords of module object at remote 0x7fffea4c2a50>, 
    args=args@entry=0x7ffff7fb9128, nargsf=<optimized out>, kwnames=kwnames@entry=0x0) at Objects/call.c:327
#4  0x000000000059e417 in _PyEval_EvalFrameDefault (tstate=0x980358 <_PyRuntime+508728>, frame=0x7ffff7fb90d0, throwflag=0) at Python/generated_cases.c.h:3773
(...)
#19 0x000000000041b78f in main (argc=<optimized out>, argv=<optimized out>) at ./Programs/python.c:15


Traceback (most recent call first):
  <built-in method meth_fastcall_keywords of module object at remote 0x7fffea4c2a50>
  File "/home/vstinner/python/main/script2.py", line 3, in foo
    _testcapi.meth_fastcall_keywords()
  File "/home/vstinner/python/main/script2.py", line 5, in bar
    foo()
  File "/home/vstinner/python/main/script2.py", line 6, in <module>
    bar()

Here the 19 frames are complete up to main().

• gcc (GCC) 13.2.1 20230728 (Red Hat 13.2.1-1)
• GNU gdb (GDB) Fedora Linux 13.2-3.fc38