Reliance on C bit fields in C API is undefined behavior

[indygreg]

BPO	45025
Nosy	@birkenfeld, @vstinner, @encukou, @methane, @serhiy-storchaka, @indygreg

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-08-27.01:47:13.094>
labels = ['type-bug', '3.8', '3.9', '3.10', '3.11', 'expert-C-API', '3.7']
title = 'Reliance on C bit fields in C API is undefined behavior'
updated_at = <Date 2021-08-31.22:02:29.674>
user = 'https://github.com/indygreg'

bugs.python.org fields:

activity = <Date 2021-08-31.22:02:29.674>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['C API']
creation = <Date 2021-08-27.01:47:13.094>
creator = 'indygreg'
dependencies = []
files = []
hgrepos = []
issue_num = 45025
keywords = []
message_count = 9.0
messages = ['400388', '400615', '400617', '400620', '400621', '400622', '400624', '400634', '400788']
nosy_count = 6.0
nosy_names = ['georg.brandl', 'vstinner', 'petr.viktorin', 'methane', 'serhiy.storchaka', 'indygreg']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45025'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

• gh-89188: replace bitfield with struct fields in PyASCIIObject #102589
• gh-89188: add PyUnicode_Data and PyUnicode_GetKind #115211
• gh-89188: Implement PyUnicode_KIND() as a function #129412

[indygreg]

At least the PyASCIIObject struct in Include/cpython/unicodeobject.h uses bit fields. Various preprocessor macros like PyUnicode_IS_ASCII() and PyUnicode_KIND() access this struct's bit field.

This is problematic because according to the C specification, the storage of bit fields is unspecified and may vary from compiler to compiler or architecture to architecture. Theoretically, a build of libpython with compiler A may not have the same storage layout of a bit field as a separate binary built with compiler B. These 2 binaries could be linked/loaded together, resulting in a crash or incorrect behavior at run-time.

https://stackoverflow.com/questions/6043483/why-bit-endianness-is-an-issue-in-bitfields/6044223#6044223

To ensure bit field behavior is consistent, the same compiler must be used for all bit field interaction. Since it is effectively impossible to ensure this for programs like Python where multiple compilers are commonly at play (a 3rd party C extension will likely not be built on the same machine that built libpython), bit fields must not be exposed in the C API. If a bit field must exist, the bit field should not be declared in a public .h header and any APIs for accessing the bit field must be implemented as compiled functions such that only a single compiler will define the bit field storage layout.

In order to avoid undefined behavior, Python's C API should avoid all use of bit fields.

This issue is in response to PyO3/pyo3#1824.