[3.13] Ensurepip fails when built with `--enable-experimental-jit` and `--with-pydebug` (Linux)

[JeffersGlass]

Bug report

Bug description:

When building for Linux/X64, the 3.13 branch will not run successfully run python -m ensurepip if built with both --enable-experimental-jit and --with-pydebug:

git checkout 3.13
make clean
./configure --enable-experimental-jit --with-pydebug
make -j8
./python -m ensurepip

This fails with an error message like:

subprocess.CalledProcessError: Command '['/home/jglass/Documents/cpython/python', '-W', 'ignore::DeprecationWarning', '-c', '\nimport runpy\nimport sys\nsys.path = [\'/tmp/tmpsu81mj6o/pip-24.0-py3-none-any.whl\'] + sys.path\nsys.argv[1:] = [\'install\', \'--no-cache-dir\', \'--no-index\', \'--find-links\', \'/tmp/tmpsu81mj6o\', \'pip\']\nrunpy.run_module("pip", run_name="__main__", alter_sys=True)\n']' died with <Signals.SIGABRT: 6>.

Building with either config flag individually successfully builds.

Results of ./python -VV: Python 3.13.0b2+ (heads/3.13:ff358616ddb, Jun 12 2024, 21:17:57) [GCC 11.4.0]

Built on Ubuntu 22.04; same failure on two separate machines. These options build successfully on main (030b452e).

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

• gh-120437: Fix _CHECK_STACK_SPACE optimization problems introduced in gh-118322 #120712
• [3.13] gh-120437: Fix _CHECK_STACK_SPACE optimization problems introduced in gh-118322 (GH-120712) #120747

[Eclips4]

Thanks for the report!
Confirmed on 3.13 branch.

[Eclips4]

Here's the full trace:

Assertion failed: (tstate->datastack_top < tstate->datastack_limit), function _PyFrame_PushUnchecked, file pycore_frame.h, line 281.
Traceback (most recent call last):
  File "/Users/admin/Projects/cpython/Lib/runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
                     "__main__", mod_spec)
  File "/Users/admin/Projects/cpython/Lib/runpy.py", line 88, in _run_code
    exec(code, run_globals)
    ~~~~^^^^^^^^^^^^^^^^^^^
  File "/Users/admin/Projects/cpython/Lib/ensurepip/__main__.py", line 5, in <module>
    sys.exit(ensurepip._main())
             ~~~~~~~~~~~~~~~^^
  File "/Users/admin/Projects/cpython/Lib/ensurepip/__init__.py", line 257, in _main
    return _bootstrap(
        root=args.root,
    ...<4 lines>...
        default_pip=args.default_pip,
    )
  File "/Users/admin/Projects/cpython/Lib/ensurepip/__init__.py", line 172, in _bootstrap
    return _run_pip([*args, "pip"], [os.fsdecode(tmp_wheel_path)])
  File "/Users/admin/Projects/cpython/Lib/ensurepip/__init__.py", line 87, in _run_pip
    return subprocess.run(cmd, check=True).returncode
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/admin/Projects/cpython/Lib/subprocess.py", line 577, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/Users/admin/Projects/cpython/python.exe', '-W', 'ignore::DeprecationWarning', '-c', '\nimport runpy\nimport sys\nsys.path = [\'/var/folders/02/ps_0fn7s0sx9kkngq6ykymdh0000gn/T/tmphjt45j3u/pip-24.0-py3-none-any.whl\'] + sys.path\nsys.argv[1:] = [\'install\', \'--no-cache-dir\', \'--no-index\', \'--find-links\', \'/var/folders/02/ps_0fn7s0sx9kkngq6ykymdh0000gn/T/tmphjt45j3u\', \'pip\']\nrunpy.run_module("pip", run_name="__main__", alter_sys=True)\n']' died with <Signals.SIGABRT: 6>.

I guess the main problem at the beginning of trace:

Assertion failed: (tstate->datastack_top < tstate->datastack_limit), function _PyFrame_PushUnchecked, file pycore_frame.h, line 281.

So it seems more related to JIT than to ensurepip.

[JeffersGlass]

Bisecting, this looks like it broke in 1ab6356.

[JeffersGlass]

This error does not appear to occur on Windows x64 builds on 3.13:

.\PCBuild\build.bat -e -d -p x64 --experimental-jit      
.\PCbuild\amd64\python_d.exe -m ensurepip      # this succeeds

.\PCbuild\amd64\python_d.exe
Python 3.13.0b2+ (heads/3.13:ffc8e21c80, Jun 13 2024, 09:01:50) [MSC v.1929 64 bit (AMD64)]   

So it seems more related to JIT than to ensurepip.

This makes sense - probably some JIT issue that ensurepip happens to be triggering.

[Zheaoli]

The bug confirm on 3.13 and has disappered in main branch

Let me find out it

[Zheaoli]

@Eclips4 would you mind assigning this to me? I may need a bit of time to find out the root cause

[Eclips4]

Sure!

[JeffersGlass]

Playing around with this and bisecting main, it looks like the commit where this was fixed is 9b8611e ... which doesn't make a lot of sense to me, that's part of a PR all about typing... Perhaps I messed up the bisect.

For what its worth, this is what I ran with git bisect run:

#!/bin/sh
make distclean -j8 || true
(./configure --enable-experimental-jit --with-pydebug && make -j8) || exit 125
! ./python -m venv venv # Negated, since the earlier commit is bad and the later one is good

[Zheaoli]

import runpy
import sys
sys.path = ['/tmp/tmp04bw2hi9/pip-23.3.2-py3-none-any.whl'] + sys.path
sys.argv[1:] = ['install', '--no-cache-dir', '--no-index', '--find-links', '/tmp/tmp04bw2hi9', 'pip']
runpy.run_module("pip", run_name="__main__", alter_sys=True)

This bug is confirmed that is introduced into the codebase in
1ab6356

I will dive more deeper, but I may need @markshannon help

[Zheaoli]

After more inspect, I think this is about Tier2, not JIT

./configure --with-pydebug --enable-pystats --enable-profiling --with-dtrace --enable-experimental-jit=interpreter

Continue to dive deeper, but I may need @Fidget-Spinner help...

BTW this issue should be labeled about Tier2 cc @Eclips4