cProfile/profile do not output results if profiling terminates because of SIGTERM

[SnoopJ]

Bug report

cProfile and profile do not produce any results if the process being profiled is terminated with SIGTERM, although the documentation for cProfile and profile and the behavior of the profiler in the case of an exception (KeyboardInterrupt in particular) do make it seem like the profiler will always write output "at the end" of profiling, and the actual behavior on this edge case is somewhat surprising as a result.

I discovered this behavior while trying to profile a worker subprocess in an application that sends SIGTERM to that subprocess during shutdown. I was able to write my own profiling helper that registers a signal handler, but thought it made sense to open an issue and ask if this is a case of slightly murky documentation, or something that can be improved in profile. Considering the complexity of signal handling, I'm guessing this is more likely to be considered a missing stair in the documentation, but I'm happy to send a PR for either case if others agree that this behavior isn't entirely obvious.

The closest issue I can find in the CPython issue history is #50228 which doesn't involve signals.

Reproduction

$ cat repro.py
import time

x = 0

while True:
    x += 1
    print(x)
    time.sleep(0.1)
$ python3.9 -m cProfile --outfile out.prof repro.py & PID=$!; sleep 0.5; kill -s SIGTERM ${PID}
[2] 1113607
1
2
3
4
[2]-  Terminated              python3 -m cProfile --outfile out.prof repro.py
$ ls out.prof
ls: cannot access 'out.prof': No such file or directory

Environment

• CPython versions tested on: 3.9, 3.8
• Operating system and architecture: Ubuntu 20.04

Linked PRs

• gh-106922: Fix error location for constructs with spaces and parentheses #108959

[sunmy2019]

The only way you handle signals (except SIGINT) from the Python side is to use the signal module.

[sunmy2019]

If this really needs a fix, the only thing we can do is register signal handlers.

But is this really needed? Pdb doesn't handle signals either.

[SnoopJ]

The only way you handle signals (except SIGINT) from the Python side is to use the signal module.

Right, this is what the helper I linked to does.

But is this really needed?

It may not be, but a note in the docs that the profiler will not leave any record if terminated by signal would have saved me some time, and I thought that I should ask about adding one.

[gaogaotiantian]

It may not be, but a note in the docs that the profiler will not leave any record if terminated by signal would have saved me some time, and I thought that I should ask about adding one.

Well SIGINT works :)

Handling common cases for SIGTERM on Linux is definitely possible, but there are a couple of considerations here:

1. SIGTERM is basically useless on Windows, so you still can't handle the case on Windows, which causes inconsistent behavior on different platforms, but that's not a deal breaker.
2. Handlers for SIGTERM could potentially be overwritten by user code. Still not a deal breaker, but even more inconsistent behavior.
3. Whether cProfile should produce output when receiving SIGTERM is debatable - depending on how you understand the signal.

So the question is not whether we can achieve this, is whether it's worth it to do it - how many new cases are covered with the new feature, how much confusion is caused by it?

cProfile does not support multiprocess by default, you'll need some wrappers anyway and you can do this in your wrappers. For the single process usage, it's not quite easy for me to imagine a common use case for this feature(compared to Ctrl+C break).

Again, I'm not opposed to the feature, just bring up some thoughts on it.

[SnoopJ]

So the question is not whether we can achieve this, is whether it's worth it to do it - how many new cases are covered with the new feature, how much confusion is caused by it?

It seems very possible that I'm the only person who is surprised/confused by this behavior. 😅 Since filing this, I've switched[1] from profile to the third-party profiler py-spy, and it turns out py-spy has the same behavior if the profiler is sent SIGTERM. Maybe it's obvious to others that the profiler just drops its data on the ground in this case.

I don't think very many "real" user cases would be covered by extending the profiler to worry about signals, and the complexity it would add is appreciable for the reasons you point out. I think I'm erring in favor of fleshing out the existing docs to make it clear what "end of the profile run" means when using -m profile or -m cProfile, if any change is worth making.

---

[1] for reasons unrelated to this issue, I realized I need a time-resolved profile, not just call/time statistics

[gaogaotiantian]

Not sure what you meant by "time resolved" profile. The major difference between the results of py-spy and cProfile is the call stack. cProfile is not able to generate call stacks by design. py-spy generates a flamegraph which still does not include data like order of events(calls) or duration of individual event.

If you are looking for a profiler that generates that kind of data(events on a timeline), supports multiprocessing/subprocess natively and handles SIGTERM, try viztracer. (Yes, it's my project).

Back to the topic, yes we can add some notes on cProfile/profile to mention that data won't be generated if it's terminated by SIGTERM, that could be helpful. Do you want to submit a PR? I can do it as well.