Public C API for accessing code object fields

[Fidget-Spinner]

A user/maintainer of a popular library documented their use case for accessing fields in the code object. In 3.11 the fields in C are gone but still available as properties in Python. Accessing these properties can be made faster by exposing their functions in the C API. The current option is PyObject_GetAttrString(code, "attr") which is much slower.

https://discuss.python.org/t/getting-the-class-name-of-a-code-frame-object-in-cpython-3-11-c-api/17396

A similar issue is #92154.

I am marking this as a release blocker @pablogsal as it would be mucho bueno if we could get this before rc1.

[pablogsal]

Mucho bueno indeed :)

I am willing to consider it but how many new APIs are we talking about? Also, we should double check and agree that exposing these fields is not going to backfire on us if we decide to change the implementation or add new stuff that will invalidate the meaning of the current fields. Maybe we can consider adding them with prefixed underscores (probably a bad idea)?

[pablogsal]

CC @markshannon @brandtbucher @iritkatriel

[Fidget-Spinner]

3 getter functions. I've thought about whether they will restrict our code. Thing is we already are restricted because all three are already exposed as Python properties. So we have no choice but to maintain backwards compatiblity, even without the new C API changes.

[markshannon]

PyObject_GetAttrString(code, "attr") may be slower, but I doubt that PyObject_GetAttr(code, attr) is much slower given that most of the attributes of code objects are computed.

Calling PyObject_GetAttr(code, co_varnames) requires that the co_varnames tuple is created. Dispatching is not likely to be any slower than creating the tuple.

If performance is the issue, we should offer a faster API (for 3.12)
Something like:
PyObject *PyCode_GetLocalName(int n)

We should also offer the same API in Python.

[markshannon]

Not only that, but co_nlocals and co_varnames don't make much sense.

def f(a, b):
     c = a + b
    def d(): 
       return c
    return d

def g(a, b):
     c = a + b
    def d(): 
       return a + b
    return d

>>> f.__code__.co_nlocals
3
>>> g.__code__.co_nlocals
4
>>> f.__code__.co_varnames
('a', 'b', 'd')
>>> g.__code__.co_varnames
('a', 'b', 'c', 'd')

[Fidget-Spinner]

If performance is the issue, we should offer a faster API (for 3.12) Something like: PyObject *PyCode_GetLocalName(int n)

We should also offer the same API in Python.

Yeah I planned to lazily create the introspection information and cache them there (kind of like how debug information in frames are lazily created). However, that missed that for 3.11 so it should go in 3.12. Something like

struct PyCodeObject {
    ...
    PyCodeIntrospectInformation *introspect_information;
}

struct debug information {
    PyObject *co_code;
    PyObject *co_varnames;
    PyObject *co_freevars;
    ...
}

[gvanrossum]

Given that the user requesting this is writing a profiler and needs access to the frame as well, why not just recommend that they include internal headers and call _PyCode_GetVarnames(), _PyCode_GetCellvars and _PyCode_GetFreevars directly? Looking at #95008 (presumably a tentative fix?) it would seem that all the required functions already exist but aren't public any more. That seems prudent since there are some concerns about the future-preparedness of these functions anyway.

In the past I think we would never have considered such a late request for new APIs a blocker.