asyncio signal handler receives signals from child processes

[ozppupbg]

Bug report

Today I ran into a very weird behaviour in one of my projects, that uses asyncio and multiprocessing.
When I interrupted the program with a Ctrl-C on the terminal, the signal handler set in asyncio with loop.add_signal_handler was triggered multiple times! Further inspection showed, that if I manually send a SIGINT to one of the worker child processes, it triggered the main process signal handler.

I could trace the issue to the socket used by asyncio to receive the signals via signal.set_wakeup_fd, which seems to receive the signal bytes also from the child processes.

Running signal.set_wakeup_fd(-1) at least once in the worker processes worked as a workaround for me but cannot be the final solution.

Also, it seems, that other signal operations also can clear the issue (for example some automatically manager child processes did not trigger the bug, as they seem to ignore SIGINT).

In my project I used concurrent.futures.ProcessPoolExecutor, but I don't think that matters, as long as the signal handler (and wakeup fd) is set before the child is started.

Your environment

• CPython versions tested on:
  Python 3.10.4
• Operating system and architecture:
  Kali Linux rolling with kernel: Debian 5.16.18-1kali1 (2022-04-01) x86_64

Bug output
asyncio_signal_bug.py.txt

# python3 asyncio_signal_bug.py
394236 Signal Handler set
394237 Worker 1 started
394240 Worker 2 started
^C394236 received signal: Signals.SIGINT
394236 received signal: Signals.SIGINT
394236 received signal: Signals.SIGINT
^C394236 received signal: Signals.SIGINT
394236 received signal: Signals.SIGINT
394236 received signal: Signals.SIGINT

# python3 asyncio_signal_bug.py --set-in-children
394241 Signal Handler set
394242 Worker 1 started
394245 Worker 2 started
^C394241 received signal: Signals.SIGINT
^C394241 received signal: Signals.SIGINT
^C394241 received signal: Signals.SIGINT
^C394241 received signal: Signals.SIGINT
^C394241 received signal: Signals.SIGINT
^C394241 received signal: Signals.SIGINT

Notice, that in the first block, for every ^C, there are three signals received, whereas in the second block (if signal.set_wakeup_fd(-1) was called in the workers) there is only one.

[shivaraj-arch]

% ~/.pyenv/versions/3.10.4/bin/python asyncio_signal_bug.py 2>/dev/null      
42747 Signal Handler set
42749 Worker 1 started
42750 Worker 2 started
^C42747 received signal: Signals.SIGINT
^C42747 received signal: Signals.SIGINT
^C42747 received signal: Signals.SIGINT
^C42747 received signal: Signals.SIGINT
^C42747 received signal: Signals.SIGINT
^C42747 received signal: Signals.SIGINT

% ~/.pyenv/versions/3.10.4/bin/python asyncio_signal_bug.py --set-in-children 2>/dev/null
42755 Signal Handler set
42757 Worker 1 started
42758 Worker 2 started
^C42755 received signal: Signals.SIGINT
^C42755 received signal: Signals.SIGINT
^C42755 received signal: Signals.SIGINT
^C42755 received signal: Signals.SIGINT
^C42755 received signal: Signals.SIGINT
^C42755 received signal: Signals.SIGINT

On Mac it runs the same as per the docs.
Python signal handlers are always executed in the main Python thread of the main interpreter, even if the signal was received in another thread. This means that signals can’t be used as a means of inter-thread communication docs
Let me know if I have left something out while testing.

[ozppupbg]

I just tried on a different system with Python 3.6.9 on Ubuntu 18.04.6 LTS with Kernel 4.15.0-188 and could confirm the buggy behaviour (after adjusting for some asyncio API naming changes).

I expect, that it has either something to do with Mac handling signalfd (the underlying syscall) slightly different or how your terminal issues SIGINT to child processes. I used Bash in my experiments, which obviously sends a SIGINT to the entire process group.

Could you please try to manually send a signal to one of the worker processes with kill -2 <worker PID>?

For me this is also received in the parent process.

Also, please note that this issue does not concern an inter-thread, but an inter-process scenario.

[shivaraj-arch]

Tried with bash 5.1. Also tried sending manual signal. The behaviour is the same ( see complete trace below without redirection to null ) with or without set children argument.
when manual kill signal is given the traceback does not spill on to terminal, there is no Process SpawnProcess-1/2 .
Maybe its difference with how Mac m1 and Linux kernels handle.

bash-5.1$ ~/.pyenv/versions/3.10.4/bin/python asyncio_signal_bug.py 
42834 Signal Handler set
42836 Worker 1 started
42837 Worker 2 started
^CProcess SpawnProcess-1:
Process SpawnProcess-2:
42834 received signal: Signals.SIGINT
Traceback (most recent call last):
  File "/Users/user1/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/user1/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/user1/.pyenv/versions/3.10.4/lib/python3.10/concurrent/futures/process.py", line 237, in _process_worker
    call_item = call_queue.get(block=True)
  File "/Users/user1/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/queues.py", line 102, in get
    with self._rlock:
  File "/Users/user1/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
Traceback (most recent call last):
  File "/Users/user1/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/user1/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/user1/.pyenv/versions/3.10.4/lib/python3.10/concurrent/futures/process.py", line 237, in _process_worker
    call_item = call_queue.get(block=True)
  File "/Users/user1/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/queues.py", line 103, in get
    res = self._recv_bytes()
  File "/Users/user1/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/connection.py", line 221, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/Users/user1/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/connection.py", line 419, in _recv_bytes
    buf = self._recv(4)
  File "/Users/user1/.pyenv/versions/3.10.4/lib/python3.10/multiprocessing/connection.py", line 384, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
^C42834 received signal: Signals.SIGINT
^C42834 received signal: Signals.SIGINT
^C42834 received signal: Signals.SIGINT
^C42834 received signal: Signals.SIGINT
^C42834 received signal: Signals.SIGINT

[ozppupbg]

I think I just found the reason:

https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

spawn

The parent process starts a fresh Python interpreter process. The child process will only inherit those resources necessary to run the process object’s run() method. In particular, unnecessary file descriptors and handles from the parent process will not be inherited. Starting a process using this method is rather slow compared to using fork or forkserver.
Available on Unix and Windows. The default on Windows and macOS.

fork

The parent process uses os.fork() to fork the Python interpreter. The child process, when it begins, is effectively identical to the parent process. All resources of the parent are inherited by the child process. Note that safely forking a multithreaded process is problematic.
Available on Unix only. The default on Unix.

If a different process start method is used, this of course changes the behaviour!

Can you try adding:
import multiprocessing as mp
and in main:
mp.set_start_method('fork')

When I set it to 'spawn' for me, I saw exectly the same behaviour you see.

[shivaraj-arch]

True , spawn vs fork is the issue. I added fork as multiprocessing start method as you mentioned and the result is exactly what you found in beginning. But I am not sure why traceback is spilling over during spawn.

bash-5.1$ ~/.pyenv/versions/3.10.4/bin/python asyncio_signal_bug.py 
42906 Signal Handler set
42907 Worker 1 started
42908 Worker 2 started
^C42906 received signal: Signals.SIGINT
42906 received signal: Signals.SIGINT
42906 received signal: Signals.SIGINT
^C42906 received signal: Signals.SIGINT
42906 received signal: Signals.SIGINT
42906 received signal: Signals.SIGINT
bash-5.1$ ~/.pyenv/versions/3.10.4/bin/python asyncio_signal_bug.py --set-in-children
42909 Signal Handler set
42910 Worker 1 started
42911 Worker 2 started
^C42909 received signal: Signals.SIGINT
^C42909 received signal: Signals.SIGINT
^C42909 received signal: Signals.SIGINT
^C42909 received signal: Signals.SIGINT
^C42909 received signal: Signals.SIGINT
^C42909 received signal: Signals.SIGINT

[ozppupbg]

But I am not sure why traceback is spilling over during spawn.

I guess, the worker processes terminate and thus print the interrupt output.

You should be able to check, if they are still running after the first ^C.

[shivaraj-arch]

Workers are terminating and interrupt is not caught on first press of ctrl-c.
Looks like there is Base Exception asyncio issue with handling KeyboardInterrupt

[ozppupbg]

Workers are terminating and interrupt is not caught on first press of ctrl-c. Looks like there is Base Exception asyncio issue with handling KeyboardInterrupt

I'm not sure, but I think this is completely unrelated to the signalfd issue I'm referring to?

As a fix my issue, I would suggest adding something along the lines of:

if hasattr(os, 'register_at_fork'):
  def close_wakeup_fd():
    signal.set_wakeup_fd(-1)
    self._signal_handlers.clear()

  os.register_at_fork(after_in_child=close_wakeup_fd)

To: https://github.com/python/cpython/blob/main/Lib/asyncio/unix_events.py _UnixSelectorEventLoop.__init__

And I just noticed, that there may be other issues regarding the sockets _ssock and _csock (from https://github.com/python/cpython/blob/7db1d2eaf367a1073191c80c7baeee41ae1f2f21/Lib/asyncio/selector_events.py BaseSelectorEventLoop) in the fork case:

If a worker uses asyncio itself and they use the same socketpair for their internal wakeup, it might get messy!

[kumaraditya303]

asyncio and fork don't work together. This seems to be a duplicate of #66285.

[renan-r-santos]

I was able to reproduce the issue using the code below both on Ubuntu 22.04 and MacOS 12.6 Intel using Python 3.10.5. Sending SIGTERM to the child also terminates the parent. Adding signal.set_wakeup_fd(-1) to the child or setting the multiprocessing start method to spawn fix the problem.

@kumaraditya303 I was reading the issue you mentioned but it doesn't look like a duplicate because the child isn't attempting to use asyncio's event loop.

import asyncio
import multiprocessing
import os
import signal
import time


def child_signal_handler(signum, _frame):
    print(f"Child signal handler received signal {signum}")
    raise SystemExit(0)


def child_main():
    signal.signal(signal.SIGTERM, child_signal_handler)
    print(f"Child started, PID: {os.getpid()}")
    while True:
        time.sleep(1)


def parent_signal_handler(signum):
    print(f"Parent signal handler received signal {signum}")
    raise SystemExit(0)


async def parent_main():
    loop = asyncio.get_running_loop()
    loop.add_signal_handler(signal.SIGTERM, lambda s=signal.SIGTERM: parent_signal_handler(s))

    multiprocessing.set_start_method("fork")
    process = multiprocessing.Process(target=child_main)
    process.start()

    while True:
        await asyncio.sleep(2)
        os.kill(process.pid, signal.SIGTERM)


if __name__ == "__main__":
    asyncio.run(parent_main())