multiprocessing.Pool(64) throws on Windows

[saschanaz]

BPO	45077
Nosy	@pfmoore, @tjguk, @zware, @eryksun, @zooba, @corona10, @saschanaz

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-09-01.13:20:44.186>
labels = ['type-bug', '3.9', '3.10', '3.11', 'library', 'OS-windows']
title = 'multiprocessing.Pool(64) throws on Windows'
updated_at = <Date 2021-10-03.11:23:58.490>
user = 'https://github.com/saschanaz'

bugs.python.org fields:

activity = <Date 2021-10-03.11:23:58.490>
actor = 'saschanaz'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'Windows']
creation = <Date 2021-09-01.13:20:44.186>
creator = 'saschanaz'
dependencies = []
files = []
hgrepos = []
issue_num = 45077
keywords = []
message_count = 3.0
messages = ['400832', '400834', '400987']
nosy_count = 7.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'corona10', 'saschanaz']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45077'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

• gh-89240: Limit multiprocessing.Pool to 61 workers on Windows #102920
• gh-89240: Add _winapi.BatchedWaitForMultipleObjects and tests #107873

[saschanaz]

Similar issue as the previous bpo-26903.

Python 3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing
>>> multiprocessing.cpu_count()
64
>>> multiprocessing.Pool(multiprocessing.cpu_count())
Exception in thread Thread-1:
Traceback (most recent call last):
<multiprocessing.pool.Pool state=RUN pool_size=64>
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\threading.py", line 973, in _bootstrap_inner
>>>     self.run()
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\pool.py", line 519, in _handle_workers
    cls._wait_for_updates(current_sentinels, change_notifier)
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\pool.py", line 499, in _wait_for_updates
    wait(sentinels, timeout=timeout)
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\connection.py", line 884, in wait
    ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\connection.py", line 816, in _exhaustive_wait
    res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 66

[saschanaz]

The argument-less instantiation also fails, which is worse.

>>> multiprocessing.Pool()
<multiprocessing.pool.Pool state=RUN pool_size=64>
>>> Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\threading.py", line 973, in _bootstrap_inner
    self.run()
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\pool.py", line 519, in _handle_workers
    cls._wait_for_updates(current_sentinels, change_notifier)
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\pool.py", line 499, in _wait_for_updates
    wait(sentinels, timeout=timeout)
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\connection.py", line 884, in wait
    ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python39\lib\multiprocessing\connection.py", line 816, in _exhaustive_wait
    res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 66

[eryksun]

See bpo-26903 for a similar problem in concurrent.futures.ProcessPoolExecutor. It was resolved by adding a limit constant, _MAX_WINDOWS_WORKERS == 61.

WaitForMultipleObjects() can wait on up to 64 object handles, but in this case 3 slots are already taken. The pool wait includes two events for its output and change-notifier queues (named pipes), plus the _winapi module always reserves a slot for the SIGINT event, even though this event is only used by waits on the main thread.

To avoid the need to limit the pool size, connection._exhaustive_wait() could be modified to combine simultaneous waits on up to 63 threads, for which each thread exhaustively populates a list of up to 64 signaled objects. I wouldn't want to modify _winapi.WaitForMultipleObjects, but the exhaustive wait should still be implemented in C, probably in the _multiprocessing extension module. A benefit of implementing _exhaustive_wait() in C is lightweight thread creation, directly with CreateThread() and a relatively small stack commit size.