Py_GIL_DISABLED not defined in "free-threaded" Windows C-API extensions

[colesbury]

Bug report

On Windows, the Py_GIL_DISABLED macro is not defined when building C API extensions with the free-threaded build of CPython 3.13 (i.e., --disable-gil). Now that biased reference counting (6dfb8fe) landed, this leads to incorrect/failed compilation of extensions on Windows ¹. The problem does not occur on Linux/macOS and other ./configure based builds because that build system defines the Py_NOGIL macro as part of pyconfig.h. On Windows, Py_GIL_DISABLED instead defined as a pre-processor definition in pyproject.props, but that's used by C-API extensions.

Among other problems, this leads to test_peg_generator failing on the nogil Windows buildbot (example failure).

I can think of two possible approaches to fixing this:

1. Modify setuptools (maybe setuptools.distutils?) to define Py_GIL_DISABLED when invoking the compiler if sysconfig.get_config_var("Py_NOGIL") is True.
2. In the CPython Windows build, generate a header file defining/undefing Py_GIL_DISABLED depending on if the build is invoked with --disable-gil or not.

The first option seems a little more straightforward to me, but there is a risk that we might encounter this same issue across multiple extension build systems. The second option seems like it would more robustly address this issue.

Linked PRs

• gh-111650: Generate pyconfig.h on Windows #112179
• gh-111650: Ensure pyconfig.h includes Py_GIL_DISABLED on Windows #112778

Footnotes

1. Modules built as part of the CPython build (i.e., standard library modules) still work ↩

[colesbury]

@zooba, I'd appreciate your feedback and suggestions about the best way to make sure that Windows C API extensions have Py_NOGIL defined when building with the free-threaded (--disable-gil) 3.13 builds.

[corona10]

Among other problems, this leads to test_peg_generator failing on the nogil Windows buildbot (example failure).

I was curious about how GHA passed the test, but I noticed that it was skipped from the GHA..
https://github.com/python/cpython/actions/runs/6695685454/job/18191835485

0:16:09 load avg: 10.69 [405/469/1] test_peg_generator skipped (resource denied) -- running (1): test_importlib (42.1 sec)
test_peg_generator skipped -- Use of the 'cpu' resource not enabled

[zooba]

I think we probably need to start modifying pyconfig.h at some stage in the build process. Eventually we should get an improved sysconfig [alternative] that can include the flag, but until then I don't think we want to make modifications in every build backend that's out there, nor do we want to force them to launch the target interpreter in order to get the right settings (which prevents cross-compilation, for example).

I'm not sure how best to do it though. Probably the ideal looks like:

• copy/patch pyconfig.h and write it to $(IntDir) (not back into the source tree)
• do rest of build referencing that directory instead of PC\
• copy patched pyconfig.h into $(OutDir)
• make sure Tools/msi and PC/layout pull it from the output directory and not the sources
• make sure the build in release-tools does the same (though I think it should happen automatically with the previous changes)

We shouldn't have to patch anything in the stdlib that finds the header, I'm pretty sure they were only in distutils. And an install layout should have it end up in the Include directory if Tools/msi and PC/layout are updated.

So there's a lot of moving parts, plus we need to figure out the best way to template the header so it can be modified as part of the build. There's not really anything built into our build system for this yet.

[colesbury]

Hi @zooba, I started implementing this and ran into a few issues. Not having a pyconfig.h in PC or Include breaks setuptools. The install layout can place the generated pyconfig.h in the installed Include directory, but you can also run pip directly from the CPython source directory without the installer. It seems like we'll run into fewer issues if we place the generated pyconfig.h in the PC directory.