test_cppext fails on Ubuntu 20.04 when _ctypes fails to build

[AbhigyanBose]

test_cppext fails when _ctypes fails to build

Issue: test_cppext fails on Ubuntu 20.04 resulting in a test failure. After using git bisect to check #92639 seems to be the PR that introduced this issue.

Note: _ctypes has always been failing to build for me, I'm not sure if its related, but it didn't cause any issues before the mentioned commit. Prior to this commit the test is not skipping, it succeeds.

Steps to replicate:

1. make clean
2. ./configure --with-pydebug
3. make -s -j14
4. ./python -m test test_cppext

Message when test fails:

0:00:00 load avg: 2.60 Run tests sequentially
0:00:00 load avg: 2.60 [1/1] test_cppext
/home/abhigyan/Documents/my-forks/cpython/build/test_python_343165æ/tempcwd/env/lib/python3.12/site-packages/pkg_resources/_vendor/pyparsing.py:87: DeprecationWarning: module 'sre_constants' is deprecated
  import sre_constants
Traceback (most recent call last):
  File "/home/abhigyan/Documents/my-forks/cpython/Lib/test/setup_testcppext.py", line 6, in <module>
    from setuptools import setup, Extension
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/abhigyan/Documents/my-forks/cpython/build/test_python_343165æ/tempcwd/env/lib/python3.12/site-packages/setuptools/__init__.py", line 18, in <module>
    from setuptools.dist import Distribution
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/abhigyan/Documents/my-forks/cpython/build/test_python_343165æ/tempcwd/env/lib/python3.12/site-packages/setuptools/dist.py", line 38, in <module>
    from setuptools import windows_support
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/abhigyan/Documents/my-forks/cpython/build/test_python_343165æ/tempcwd/env/lib/python3.12/site-packages/setuptools/windows_support.py", line 2, in <module>
    import ctypes
    ^^^^^^^^^^^^^
  File "/home/abhigyan/Documents/my-forks/cpython/Lib/ctypes/__init__.py", line 8, in <module>
    from _ctypes import Union, Structure, Array
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '_ctypes'
test test_cppext failed -- Traceback (most recent call last):
  File "/home/abhigyan/Documents/my-forks/cpython/Lib/test/test_cppext.py", line 25, in test_build
    self._test_build()
    ^^^^^^^^^^^^^^^^^^
  File "/home/abhigyan/Documents/my-forks/cpython/Lib/test/test_cppext.py", line 56, in _test_build
    self.fail(f"Build failed with exit code {proc.returncode}")
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Build failed with exit code 1

test_cppext failed (1 failure)

== Tests result: FAILURE ==

1 test failed:
    test_cppext

Total duration: 10.6 sec
Tests result: FAILURE

Message when building:

Note: Deepfreeze may have added some global objects,
      so run 'make regen-global-objects' if necessary.
 CC='gcc -pthread' LDSHARED='gcc -pthread -shared    ' OPT='-g -Og -Wall' 	./python -E ./setup.py -q build
/home/abhigyan/Documents/my-forks/cpython/Modules/_ctypes/_ctypes.c:118:10: fatal error: ffi.h: No such file or directory
  118 | #include <ffi.h>
      |          ^~~~~~~
compilation terminated.

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_bz2                  _curses               _curses_panel      
_dbm                  _gdbm                 _lzma              
_tkinter              _uuid                 readline           
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py have not
been built, they are *disabled* by configure:
_sqlite3                                                       


Failed to build these modules:
_ctypes                                                        

renaming build/scripts-3.12/pydoc3 to build/scripts-3.12/pydoc3.12
renaming build/scripts-3.12/idle3 to build/scripts-3.12/idle3.12
renaming build/scripts-3.12/2to3 to build/scripts-3.12/2to3-3.12

My environment

• CPython versions tested on: 3.12 (main branch from github)
• Operating system and architecture: Ubuntu 20.04, 64 Bit

[vstinner]

test_cppext fails when _ctypes fails to build

test_cppext now require a working venv command. setuptools uses ctypes.

Does test_venv pass? test_venv skips a test if ctypes is missing:

    # Issue #26610: pip/pep425tags.py requires ctypes
    @unittest.skipUnless(ctypes, 'pip requires ctypes')
    @requires_zlib()
    def test_with_pip(self):
        self.do_test_with_pip(False)
        self.do_test_with_pip(True)

Maybe I need to add a decorator to test.support to skip a test if venv+pip is known to not work.

[AbhigyanBose]

@vstinner

test_venv is Successful overall but skips test_with_pip, as you mentioned, due to 'pip requires ctypes'. All other OS relevant tests are passing.

[vstinner]

test_cppext fails when _ctypes fails to build

I created #92844 to fix it.

[vstinner]

Since setuptools 61.0, ctypes is no longer imported by setuptools.windows_support at startup: pypa/setuptools@a48561e

[AbhigyanBose]

Issue fixed by #92844