PyArg_ParseTupleAndKeywords() and non-ASCII keyword names

[serhiy-storchaka]

Most of C strings in the C API are implied to be UTF-8 encoded. PyArg_ParseTupleAndKeywords() mostly works with non-ASCII keyword names as they are UTF-8 encoded. Except one case, when you pass argument by keyword with invalid non-ASCII name to a function that has optional parameter with non-ASCII UTF-8 encoded name. In this case you get a crash in the debug build.

It was caused by combination of f4934ea and a83a6a3 (bpo-28701, #72887). Before these changes you simply got inaccurate or even wrong error message.

Examples:

1. Parameters: "ä"
   Keyword arguments: "ë"
   Old behavior: TypeError "'ë' is an invalid keyword argument for this function"
   Current behavior: crash
   Expected behavior: TypeError "'ë' is an invalid keyword argument for this function"

2. Parameters: "ä"
   Keyword arguments: "ä"
   Old behavior: TypeError "invalid keyword argument for this function"
   Current behavior: crash
   Expected behavior: TypeError "'ä' is an invalid keyword argument for this function"

3. Parameters: "ä", "ë"
   Keyword arguments: "ä", "ë"
   Old behavior: TypeError "'ë' is an invalid keyword argument for this function"
   Current behavior: crash
   Expected behavior: TypeError "'ä' is an invalid keyword argument for this function"

In case 1 the pre-bpo-28701 behavior was correct, in case 2 it was correct but not precise (it failed to find the name of invalid keyword argument), in case 3 it was wrong (it found wrong name). In all cases there is a crash currently.

Linked PRs

• gh-110815: Support non-ASCII keyword names in PyArg_ParseTupleAndKeywords() #110816
• gh-110815: Improve tests for PyArg_ParseTupleAndKeywords() #110817
• [3.12] gh-110815: Improve tests for PyArg_ParseTupleAndKeywords() (GH-110817) #110825

[serhiy-storchaka]

I am not sure how to classify this issue: as a bug or a new feature. On one hand, it was a regression in 3.7, crash is worse than wrong error message. On other hand, it was not explicitly documented that non-ASCII names are supported, there were no tests, and if it worked, it was only by accident. And it was a long time ago. So, official support of non-ASCII names can be a new feature.