Exception in ucs2lib_utf8_encoder in _bootstrap_python

[pgy]

Bug report

I am building python in a directory that has non-ascii name on linux and get the following error:

./_bootstrap_python ./Programs/_freeze_module.py abc ./Lib/abc.py Python/frozen_modules/abc.h
Exception ignored error evaluating path:
Traceback (most recent call last):
  File "<frozen getpath>", line 349, in <module>
ModuleNotFoundError: No module named 'encodings'
Fatal Python error: error evaluating path
Python runtime state: core initialized

Current thread 0x00007f16d6c10740 (most recent call first):
  <no Python frame>
make: *** [Makefile:1218: Python/frozen_modules/abc.h] Error 1

The problem seems to be that the limited environment used to execute the precompiled Modules/getpath.py does not have the encodings module.

I tracked the exception to a PyImport_ImportModule("encodings"); call in Python/codecs.c which is a consequence of a PyCodec_LookupError that happens in ucs2lib_utf8_encoder which I think gets called in getpath_dirname.

The original string is /home/pgy/letöltések/cpython/_bootstrap_python, and the unicode object ucs2lib_utf8_encoder gets as an argument is:

(rr) p PyObject_Print(unicode, stderr, 0)
'/home/pgy/let\udcc3\udcb6lt\udcc3\udca9sek/cpython/_bootstrap_python'

Your environment

• CPython versions tested on: 7db1d2e
• Operating system and architecture: Linux hostname 5.15.44-1-lts #1 SMP Mon, 30 May 2022 13:45:47 +0000 x86_64 GNU/Linux

[vstinner]

I created PR #97645 to fix the root issue.

I can reproduce the issue:

• Clone Python Git repository in a non-ASCII directory: /home/vstinner/python/mainé (é is non-ASCII)
• Build Python: ./configure --with-pydebug && make
• make fails with: ModuleNotFoundError: No module named 'encodings'

My locale encoding is UTF-8, my LC_CTYPE locale is fr_FR.UTF-8.

The problem is that the getpath_dirname() function of Modules/getpath.c encodes the Unicode path to UTF-8/strict using s format and PyArg_ParseTuple(), whereas a path can contain surrogate characters.

The getpath_dirname() and getpath_basename() convert their Unicode input string to bytes just to be able to use strrchr(path, SEP) since SEP is a bytes string. There is no need to convert these strings to bytes, whereas the conversion is the root issue here.

[vstinner]

Fixed by PR #97645. Backport to 3.11: PR #97677.