Getting the class name of a code/frame object in CPython 3.11 C API

Python Help
1

Hello there.

I maintain a library called pyinstrument, a profiler for CPython. It observes a program’s execution and prints output that looks like this-

  _     ._   __/__   _ _  _  _ _/_   Recorded: 14:42:13  Samples:  30
 /_//_/// /_\ / //_// / //_'/ //     Duration: 0.332     CPU time: 0.050
/   _/                      v4.2.0

Program: examples/wikipedia_article_word_count.py

0.332 <module>  <string>:1
   [9 frames hidden]  <string>, runpy, <built-in>
      0.330 _run_code  runpy.py:63
      └─ 0.330 <module>  wikipedia_article_word_count.py:1
         ├─ 0.281 main  wikipedia_article_word_count.py:39
         │  └─ 0.278 download  wikipedia_article_word_count.py:15
         │     ├─ 0.274 urlopen  urllib/request.py:139
         │     │     [61 frames hidden]  urllib, http, socket, ssl, <built-in>...
         │     └─ 0.004 read  http/client.py:449
         │           [12 frames hidden]  http, socket, ssl, <built-in>
         └─ 0.049 <module>  urllib/request.py:1
               [39 frames hidden]  urllib, hashlib, <built-in>, http, ss...

I’m currently working on a feature that prints the class name of a function, if that function is a method. (e.g. in the above output, it would print HTTPResponse.read, rather than just read). It does this by looking at the first argument of the frame and, if it’s called ‘self’ or ‘cls’, it gets the type of this object.

That code can be seen here: pyinstrument/stat_profile.c at e4a6e0805a30f0de0077534ac87d1cfeee53beb4 · joerick/pyinstrument · GitHub . It uses code->co_varnames and frame->f_localsplus to read this information. It’s slightly complicated by the fact that the local might be a ‘cell’ variable, but that’s handled too.

(It’s quite important that the technique used is fast. Keeping the profiler low-overhead is important because otherwise it can distort the data. So performance is a concern.)

I’ve upgraded this branch to CPython 3.11, and I’ve now got a problem. I can’t access these fields any more, as they’ve become private to the interpreter. The release notes say “f_localsplus: no public API (renamed to f_frame.localsplus)”. co_varnames isn’t mentioned in the release notes, but it’s gone from the headers.

The only way I can see to work around this is using the new PyFrame_GetLocals function. But that would have the side effect of calling ‘fast-to-locals’ on every frame in the program - which is something that I want to avoid as it seems like it could have pretty major performance impacts - adding profiling overhead or just changing how the program performs.

So my question is - is there a better way to get the class name of a code/frame object? The variable named ‘self’/‘cls’ is a bit of a hack anyway. Perhaps there’s a static way that just uses the ‘code’ object? Or, is the ‘fast-to-locals’ thing not as much of an issue as I’m making it?

2

Hi Joe,

First things first: if public fields have suddenly been made private without warning, that is a breaking change that should be reported on the bug tracker.

As for reporting the qualified name of the method, you should look at the __qualname__ attribute on the function object:


from http.client import HTTPResponse

print(HTTPResponse.read.__qualname__)

# --> prints 'HTTPResponse.read'

If you have the bound method object instead:


import http.client

connection = http.client.HTTPSConnection("www.python.org")

connection.request("GET", "/")

response = connection.getresponse()

print(response.read.__func__.__qualname__)

# --> prints 'HTTPResponse.read'

Does this help?

1 Like
3

Regarding the code objects:
One temporary workaround for any of the code objects is to access them from the Python level. E.g. PyObject_GetAttrString(code, "co_varnames"). However, this is slower than accessing via the C API.

Currently there is only a public C API getter for accessing the co_code attribute (PyCode_GetCode). I’ll try to add function getters to the C API for the other fields before the next 3.11 release. These functions will likely be slower than accessing the struct fields directly in 3.10. In 3.11 a lot of these fields are gone. We still maintain compatibility at the Python level by reconstructing the pre-3.11 fields on the fly. This means you should try to reduce accesses to these fields because they’re now almost always dynamically created.

Regarding the frame objects, I’m not too sure. Maybe @vstinner can help you. Should we add a public C API for those?

1 Like
4

At the risk of speaking out of turn, @joerick, have you considered including internal headers and call _PyCode_GetVarnames() , _PyCode_GetCellvars and _PyCode_GetFreevars directly?

I agree it’s not pretty, but since you’re writing a profiler you’re clearly already depending on unstable APIs.

2 Likes
5

Thanks all, this is super helpful! I’ll do some profiling on the __qualname__ approach to see if that is quick enough to be workable. PyObject_GetAttrString looks convenient too.

I’ll try to add function getters to the C API for the other fields before the next 3.11 release. These functions will likely be slower than accessing the struct fields directly in 3.10. In 3.11 a lot of these fields are gone. We still maintain compatibility at the Python level by reconstructing the pre-3.11 fields on the fly. This means you should try to reduce accesses to these fields because they’re now almost always dynamically created.

Thanks, that would be useful, and might help others who are using these objects from the C API side.

have you considered including internal headers and call _PyCode_GetVarnames() , _PyCode_GetCellvars and _PyCode_GetFreevars directly?

I had not realised this was possible! So those symbols are still accessible to a C extension, they’re just in an internal header? That’s very cool. I’ll have a play with this. If the above routes are not feasible, dropping to this level sounds like a good backstop :slight_smile:

6

I proposed adding PyFrame_GetVar(), but the discussion didn’t go far: [C API] Add PyFrame_GetVar(frame, name) function · Issue #91248 · python/cpython · GitHub

It seems like Mark plans to do something more generic, but I didn’t understand the details.

7

See also gh-94936: C getters: co_varnames, co_cellvars, co_freevars by Fidget-Spinner · Pull Request #95008 · python/cpython · GitHub : would such API fills your needs?

8

Thank you for merging the code getters PR. That will help for sure. I haven’t had a chance to fully test it yet, but I think I can use the new PyCode_GetVarnames getter to check for a self varname, and then if it’s there, use PyFrame_GetLocals and PyDict_GetItem to find self and get the type with _PyType_Name.

That still has the side effect of doing fast-to-locals on a few frames, but at least it’s only the frames that actually have self arguments.

PyFrame_GetVar() would be really nice, yeah! Especially from a performance PoV, if it could avoid all the object creation that the above involves (tuple for the code args and dict for the locals).