Python `3.11` frame structure and various changes

_PyFrame_GetGenerator() takes a _PyInterpreterFrame , are you not getting a
compilation error passing a PyFrameObject
?

TBH, I have not tried using it yet. I just seen similar code in ceval.c and asked if this is the right way to do it. See yappi/yappi/_yappi.c at 0fd2b5f9b2885f7fd518e6a42c8ee4ab461c5f96 · sumerc/yappi · GitHub. If you have any idea on how to do it, I would love to hear :slight_smile:

While this is all still fresh in your memory, would you mind briefly writing up
where you you need to delve into CPython internals. It would be really valuable.
We are trying to flesh out the API for the sort of introspection you are doing.

Sure. Would love to do that. I am writing what I need and WHY I need it for. That is because: maybe someone can suggest a better way(s) to do it.

1/
PyFrameObject->f_state: I use this to differentiate a await and an exit. I need this distinction to measure wall-time of a specific coroutine. E.g., I did not increment function call count when function is yielded for an await. And currently, I am not sure on how to do it.

2/
I use frame->co_flags & CO_COROUTINE || frame->co_flags & CO_ITERABLE_COROUTINE || frame->co_flags & CO_ASYNC_GENERATOR to determine if the running function is simply async or not.

3/
frame->co_varnames The reason I need this is the same reason with the thread opened here Get the class name if the function is a method. __qualname__ was not always possible before (2.7 times…), thus I did not change the implementation. There was also another issue on not using qualname (maybe performance), but I honestly do not remember…

4/
frame->f_code->co_flags & CO_VARARGS and max_arg_count = frame->f_code->co_argcount; Capture arguments from the function arguments (by name or by positional index). We are using that in Blackfire profiler EXTENSIVELY. We have a metric system where we collect data whenever a specific function call happens:

e.g., collect the argument query whenever function name is equals to something:

db.execute(query='select * ...').

If the function is built-in(C function) we read it from frame->f_valuestack and if the function is a normal Python function we use: arg_name = PyTuple_GetItem(frame->f_code->co_varnames, int_arg_id-1);

5/
(_pyctx_t *)PyThreadState_GET()->context; I need this for async. profiling. Internally, every context is hold in another slot in a hash table where I aggregate measurements. From profiler’s perspective every context has a separate callstack so they need to be measured separately.

These are things I remember for now.

One question. if you don’t mind: Are you going to implement a new API to access these fields only or are you also going to restrict access to them as well?

2 Likes